Gather-scatter communication using OpenSHMEM one-sided puts with per-rank signaling for completion (OpenSHMEM 1.5). More...
Public Member Functions | |
| procedure, pass(this) | init (this, send_pe, recv_pe) |
| Initialise OpenSHMEM based communication method. | |
| procedure, pass(this) | free (this) |
| Deallocate OpenSHMEM based communication method. | |
| procedure, pass(this) | nbsend (this, u, n, deps, strm) |
| Pack the gathered shared dofs into the symmetric send buffer and issue non-blocking puts with signaling to each neighbor's recv buffer. Before each put, wait for the receiver's ack of our previous round so we never overwrite a buffer the receiver hasn't consumed yet. | |
| procedure, pass(this) | nbrecv (this) |
| No-op: receives are completed via remote put-with-signal. | |
| procedure, pass(this) | nbwait (this, u, n, op, strm) |
| Wait per-neighbor for the signal indicating that data has landed, apply the gather-scatter operation from the recv buffer into u, and ack the sender so they may overwrite the buffer in the next round. | |
| procedure(gs_comm_init), deferred, pass | init gs_comm_init |
| procedure(gs_comm_free), deferred, pass | free gs_comm_free |
| procedure(gs_nbsend), deferred, pass | nbsend gs_nbsend |
| procedure(gs_nbrecv), deferred, pass | nbrecv gs_nbrecv |
| procedure(gs_nbwait), deferred, pass | nbwait gs_nbwait |
| procedure, pass(this) | init_dofs (this) |
| procedure, pass(this) | free_dofs (this) |
| procedure, pass(this) | init_order (this, send_pe, recv_pe) |
| Obtains which ranks to send and receive data from. | |
| procedure, pass(this) | free_order (this) |
Public Attributes | |
| type(gs_shmem_buf_t) | send_buf |
| type(gs_shmem_buf_t) | recv_buf |
| type(c_ptr) | data_signals_ptr = C_NULL_PTR |
| type(c_ptr) | ack_signals_ptr = C_NULL_PTR |
| integer(kind=i8) | iter = 0 |
| type(stack_i4_t), dimension(:), allocatable | send_dof |
| A list of stacks of dof indices local to this process to send to rank_i. | |
| type(stack_i4_t), dimension(:), allocatable | recv_dof |
| recv_dof(rank_i) is a stack of dof indices local to this process to receive from rank_i. size(recv_dof) == pe_size | |
| integer, dimension(:), allocatable | send_pe |
| Array of ranks that this process should send to. | |
| integer, dimension(:), allocatable | recv_pe |
| array of ranks that this process will receive messages from | |
Detailed Description
Each PE allocates symmetric send and recv buffers sized to the global maximum number of dofs (so puts can target a well-defined offset on the remote PE's recv buffer) plus two symmetric uint64 arrays of size pe_size, indexed by the remote PE's rank: data_signals[r] holds an "iter" value written by PE r when it has put data into our recv buffer (via shmem_putmem_signal_nbi); ack_signals[r] holds an "iter" value written by PE r after it has consumed our most recent put (via shmem_uint64_atomic_set). The sender waits on its own ack_signals[dst] before each put so a fast sender cannot overwrite a buffer the receiver hasn't consumed.
Per-rank slot indexing avoids the slot-exchange handshake that a neighbor-list-position scheme would require, which is fragile because send_pe and recv_pe are pushed independently in gs_schedule and may differ in size and ordering.
Definition at line 91 of file gs_shmem.F90.
Member Function/Subroutine Documentation
◆ free() [1/2]
Definition at line 109 of file gs_shmem.F90.
◆ free() [2/2]
Definition at line 61 of file gs_comm.f90.
◆ free_dofs()
|
inherited |
Definition at line 66 of file gs_comm.f90.
◆ free_order()
|
inherited |
Definition at line 68 of file gs_comm.f90.
◆ init() [1/2]
| procedure, pass(this) gs_shmem::gs_shmem_t::init | ( | class(gs_shmem_t), intent(inout) | this, |
| type(stack_i4_t), intent(inout) | send_pe, | ||
| type(stack_i4_t), intent(inout) | recv_pe | ||
| ) |
Definition at line 108 of file gs_shmem.F90.
◆ init() [2/2]
Definition at line 60 of file gs_comm.f90.
◆ init_dofs()
|
inherited |
Definition at line 65 of file gs_comm.f90.
◆ init_order()
|
inherited |
- Parameters
-
send_pe,only contains rank ids this porcesss should send to recv_pe,only the ranks this process should receive from
Definition at line 67 of file gs_comm.f90.
◆ nbrecv() [1/2]
Definition at line 111 of file gs_shmem.F90.
◆ nbrecv() [2/2]
Definition at line 63 of file gs_comm.f90.
◆ nbsend() [1/2]
| procedure, pass(this) gs_shmem::gs_shmem_t::nbsend | ( | class(gs_shmem_t), intent(inout) | this, |
| real(kind=rp), dimension(n), intent(inout) | u, | ||
| integer, intent(in) | n, | ||
| type(c_ptr), intent(inout) | deps, | ||
| type(c_ptr), intent(inout) | strm | ||
| ) |
Definition at line 110 of file gs_shmem.F90.
◆ nbsend() [2/2]
Definition at line 62 of file gs_comm.f90.
◆ nbwait() [1/2]
| procedure, pass(this) gs_shmem::gs_shmem_t::nbwait | ( | class(gs_shmem_t), intent(inout) | this, |
| real(kind=rp), dimension(n), intent(inout) | u, | ||
| integer, intent(in) | n, | ||
| integer | op, | ||
| type(c_ptr), intent(inout) | strm | ||
| ) |
Definition at line 112 of file gs_shmem.F90.
◆ nbwait() [2/2]
Definition at line 64 of file gs_comm.f90.
Member Data Documentation
◆ ack_signals_ptr
| type(c_ptr) gs_shmem::gs_shmem_t::ack_signals_ptr = C_NULL_PTR |
Definition at line 101 of file gs_shmem.F90.
◆ data_signals_ptr
| type(c_ptr) gs_shmem::gs_shmem_t::data_signals_ptr = C_NULL_PTR |
Definition at line 97 of file gs_shmem.F90.
◆ iter
Definition at line 106 of file gs_shmem.F90.
◆ recv_buf
| type(gs_shmem_buf_t) gs_shmem::gs_shmem_t::recv_buf |
Definition at line 93 of file gs_shmem.F90.
◆ recv_dof
|
inherited |
Definition at line 52 of file gs_comm.f90.
◆ recv_pe
|
inherited |
Definition at line 58 of file gs_comm.f90.
◆ send_buf
| type(gs_shmem_buf_t) gs_shmem::gs_shmem_t::send_buf |
Definition at line 92 of file gs_shmem.F90.
◆ send_dof
|
inherited |
Definition at line 49 of file gs_comm.f90.
◆ send_pe
|
inherited |
- Note
- : this will usually be fewer than the total number of ranks size(send_pe) <= pe_size
Definition at line 56 of file gs_comm.f90.
The documentation for this type was generated from the following file:
- /home/runner/work/neko/neko/src/gs/gs_shmem.F90
