Gather-scatter communication using MPI. More...

Inheritance diagram for gs_mpi::gs_mpi_t:

Collaboration diagram for gs_mpi::gs_mpi_t:

[legend]

Public Member Functions
procedure, pass(this)	init (this, send_pe, recv_pe)
	Initialise MPI based communication method See gs_comm.f90 for details.

procedure, pass(this)	free (this)
	Deallocate MPI based communication method.

procedure, pass(this)	nbsend (this, u, n, tag, deps, strm)
	See gs_comm.f90 for more details on these routines.

procedure, pass(this)	nbrecv (this, tag)
	Post non-blocking receive operations.

procedure, pass(this)	nbwait (this, u, n, op, strm)
	Wait for non-blocking operations.

procedure, pass(this)	init_vec (this)
	Allocate the fused vector exchange buffers, sized for GS_VEC_NC components. Deferred to the first fused exchange, see gs_comm_t.

procedure, pass(this)	nbsend_vec (this, u, n, nc, tag, deps, strm)
	Post non-blocking sends for a fused nc-component exchange.

procedure, pass(this)	nbrecv_vec (this, tag, nc)
	Post non-blocking receives for a fused nc-component exchange.

procedure, pass(this)	nbwait_vec (this, u, n, nc, op, strm)
	Wait for a fused nc-component exchange and reduce each received slab.

procedure(gs_comm_init), deferred, pass	init gs_comm_init

procedure(gs_comm_free), deferred, pass	free gs_comm_free

procedure(gs_nbsend), deferred, pass	nbsend gs_nbsend

procedure(gs_nbrecv), deferred, pass	nbrecv gs_nbrecv

procedure(gs_nbwait), deferred, pass	nbwait gs_nbwait

procedure, pass(this)	init_dofs (this)

procedure, pass(this)	free_dofs (this)

procedure, pass(this)	init_order (this, send_pe, recv_pe)
	Obtains which ranks to send and receive data from.

procedure, pass(this)	free_order (this)

procedure, pass(this)	take_schedule (this, src)
	Take over the gather-scatter schedule (dof lists and peer order) of src, avoiding a second (expensive) pass over the connectivity. The data is moved rather than copied, so src is left without a schedule and must not be used for communication afterwards (it can still be freed). No communication resources are set up here; complete the handover with init_schedule once src has been freed, so that the two backends never hold their resources at the same time.

procedure, pass(this)	init_schedule (this)
	Set up this communication method for the schedule taken over by take_schedule. Collective, as init is.

procedure, pass(this)	init_vec (this)
	Fused vector halo exchange. Default implementations abort; backends that set vec_supported = .true. override them.

procedure, pass(this)	nbsend_vec (this, u, n, nc, tag, deps, strm)
	Default fused vector send. Abort unless a backend overrides it.

procedure, pass(this)	nbrecv_vec (this, tag, nc)
	Default fused vector receive. Abort unless a backend overrides it.

procedure, pass(this)	nbwait_vec (this, u, n, nc, op, strm)
	Default fused vector wait/reduce. Abort unless a backend overrides it.

Public Attributes
real(kind=rp), dimension(:), allocatable	send_buf
	Concatenated send slabs, one per peer.

real(kind=rp), dimension(:), allocatable	recv_buf
	Concatenated recv slabs, one per peer.

integer, dimension(:), allocatable	send_len
	Number of dofs to send to / receive from each peer.

integer, dimension(:), allocatable	recv_len

integer, dimension(:), allocatable	send_offset
	0-based offsets into send_buf / recv_buf for each peer.

integer, dimension(:), allocatable	recv_offset

type(mpi_request), dimension(:), allocatable	send_request
	Per-peer non-blocking MPI requests.

type(mpi_request), dimension(:), allocatable	recv_request

integer, dimension(:), allocatable	recv_indices
	Scratch arrays for MPI_Testsome on the recv side: indices of the requests completed by the call and their statuses.

type(mpi_status), dimension(:), allocatable	recv_statuses

integer	ncompleted

real(kind=rp), dimension(:), allocatable	send_buf_v
	Fused vector (multi-component) send/recv slabs, sized for up to GS_VEC_NC components. Each peer slab holds nc consecutive component blocks; the scalar send_len/offset arrays are reused, scaled by nc.

real(kind=rp), dimension(:), allocatable	recv_buf_v

type(stack_i4_t), dimension(:), allocatable	send_dof
	A list of stacks of dof indices local to this process to send to rank_i.

type(stack_i4_t), dimension(:), allocatable	recv_dof
	recv_dof(rank_i) is a stack of dof indices local to this process to receive from rank_i. size(recv_dof) == pe_size

integer, dimension(:), allocatable	send_pe
	Array of ranks that this process should send to.

integer, dimension(:), allocatable	recv_pe
	array of ranks that this process will receive messages from

logical	vec_supported = .false.
	Whether this backend implements the fused vector (multi-component) halo exchange (nbsend_vec/nbrecv_vec/nbwait_vec). When .false., the gs_op_r3 caller falls back to nc independent scalar exchanges.

logical	vec_ready = .false.
	Whether the buffers the fused vector exchange needs are in place. They are sized GS_VEC_NC times the halo, quadrupling what the backend holds, and only gs_op_r3 ever touches them, so a backend that can allocate them on its own defers that to the first fused exchange (see init_vec) rather than paying for it in every run. A backend whose vector buffers are part of an allocation the whole run has to agree on – symmetric memory, coarrays, registered memory, an RMA window – cannot defer, since a rank with no shared dofs never reaches the first fused exchange; those allocate in init and set this there.

Detailed Description

Definition at line 49 of file gs_mpi.f90.

Member Function/Subroutine Documentation

◆ free() [1/2]

procedure, pass(this) gs_mpi::gs_mpi_t::free ( class(gs_mpi_t), intent(inout) this )

Definition at line 72 of file gs_mpi.f90.

◆ free() [2/2]

procedure(gs_comm_free), deferred, pass gs_comm::gs_comm_t::free

pure virtualinherited

Definition at line 82 of file gs_comm.f90.

◆ free_dofs()

procedure, pass(this) gs_comm::gs_comm_t::free_dofs ( class(gs_comm_t), intent(inout) this )

inherited

Definition at line 87 of file gs_comm.f90.

◆ free_order()

procedure, pass(this) gs_comm::gs_comm_t::free_order ( class(gs_comm_t), intent(inout) this )

inherited

Definition at line 89 of file gs_comm.f90.

◆ init() [1/2]

procedure, pass(this) gs_mpi::gs_mpi_t::init	(	class(gs_mpi_t), intent(inout)	this,
		type(stack_i4_t), intent(inout)	send_pe,
		type(stack_i4_t), intent(inout)	recv_pe
	)

Definition at line 71 of file gs_mpi.f90.

◆ init() [2/2]

procedure(gs_comm_init), deferred, pass gs_comm::gs_comm_t::init

pure virtualinherited

Definition at line 81 of file gs_comm.f90.

◆ init_dofs()

procedure, pass(this) gs_comm::gs_comm_t::init_dofs ( class(gs_comm_t), intent(inout) this )

inherited

Definition at line 86 of file gs_comm.f90.

◆ init_order()

procedure, pass(this) gs_comm::gs_comm_t::init_order	(	class(gs_comm_t), intent(inout)	this,
		type(stack_i4_t), intent(inout)	send_pe,
		type(stack_i4_t), intent(inout)	recv_pe
	)

inherited

Parameters

send_pe,only	contains rank ids this porcesss should send to
recv_pe,only	the ranks this process should receive from

Definition at line 88 of file gs_comm.f90.

◆ init_schedule()

procedure, pass(this) gs_comm::gs_comm_t::init_schedule ( class(gs_comm_t), intent(inout) this )

inherited

Definition at line 91 of file gs_comm.f90.

◆ init_vec() [1/2]

procedure, pass(this) gs_comm::gs_comm_t::init_vec ( class(gs_comm_t), intent(inout) this )

inherited

Note: Rank local by contract: a rank with no shared dofs skips the fused exchange entirely, so an override must not communicate.

Definition at line 94 of file gs_comm.f90.

◆ init_vec() [2/2]

procedure, pass(this) gs_mpi::gs_mpi_t::init_vec ( class(gs_mpi_t), intent(inout) this )

Definition at line 77 of file gs_mpi.f90.

◆ nbrecv() [1/2]

procedure, pass(this) gs_mpi::gs_mpi_t::nbrecv	(	class(gs_mpi_t), intent(inout)	this,
		integer, intent(in)	tag
	)

Definition at line 75 of file gs_mpi.f90.

◆ nbrecv() [2/2]

procedure(gs_nbrecv), deferred, pass gs_comm::gs_comm_t::nbrecv

pure virtualinherited

Definition at line 84 of file gs_comm.f90.

◆ nbrecv_vec() [1/2]

procedure, pass(this) gs_comm::gs_comm_t::nbrecv_vec	(	class(gs_comm_t), intent(inout)	this,
		integer, intent(in)	tag,
		integer, intent(in)	nc
	)

inherited

Definition at line 96 of file gs_comm.f90.

◆ nbrecv_vec() [2/2]

procedure, pass(this) gs_mpi::gs_mpi_t::nbrecv_vec	(	class(gs_mpi_t), intent(inout)	this,
		integer, intent(in)	tag,
		integer, intent(in)	nc
	)

Definition at line 79 of file gs_mpi.f90.

◆ nbsend() [1/2]

procedure, pass(this) gs_mpi::gs_mpi_t::nbsend	(	class(gs_mpi_t), intent(inout)	this,
		real(kind=rp), dimension(n), intent(inout)	u,
		integer, intent(in)	n,
		integer, intent(in)	tag,
		type(c_ptr), intent(inout)	deps,
		type(c_ptr), intent(inout)	strm
	)

Definition at line 74 of file gs_mpi.f90.

◆ nbsend() [2/2]

procedure(gs_nbsend), deferred, pass gs_comm::gs_comm_t::nbsend

pure virtualinherited

Definition at line 83 of file gs_comm.f90.

◆ nbsend_vec() [1/2]

procedure, pass(this) gs_comm::gs_comm_t::nbsend_vec	(	class(gs_comm_t), intent(inout)	this,
		real(kind=rp), dimension(nc*n), intent(inout)	u,
		integer, intent(in)	n,
		integer, intent(in)	nc,
		integer, intent(in)	tag,
		type(c_ptr), intent(inout)	deps,
		type(c_ptr), intent(inout)	strm
	)

inherited

Parameters

u	compact shared buffer, component-outer: u((c-1)*n + idx)
n	number of shared dofs (per component)
nc	number of components

Definition at line 95 of file gs_comm.f90.

◆ nbsend_vec() [2/2]

procedure, pass(this) gs_mpi::gs_mpi_t::nbsend_vec	(	class(gs_mpi_t), intent(inout)	this,
		real(kind=rp), dimension(nc*n), intent(inout)	u,
		integer, intent(in)	n,
		integer, intent(in)	nc,
		integer, intent(in)	tag,
		type(c_ptr), intent(inout)	deps,
		type(c_ptr), intent(inout)	strm
	)

Parameters

u	compact shared buffer, component-outer: u((c-1)*n + idx). Peer i's slab holds nc consecutive blocks of send_len(i) values, so the scalar offsets/lengths are reused scaled by nc.

Definition at line 78 of file gs_mpi.f90.

◆ nbwait() [1/2]

procedure, pass(this) gs_mpi::gs_mpi_t::nbwait	(	class(gs_mpi_t), intent(inout)	this,
		real(kind=rp), dimension(n), intent(inout)	u,
		integer, intent(in)	n,
		integer	op,
		type(c_ptr), intent(inout)	strm
	)

Definition at line 76 of file gs_mpi.f90.

◆ nbwait() [2/2]

procedure(gs_nbwait), deferred, pass gs_comm::gs_comm_t::nbwait

pure virtualinherited

Definition at line 85 of file gs_comm.f90.

◆ nbwait_vec() [1/2]

procedure, pass(this) gs_comm::gs_comm_t::nbwait_vec	(	class(gs_comm_t), intent(inout)	this,
		real(kind=rp), dimension(nc*n), intent(inout)	u,
		integer, intent(in)	n,
		integer, intent(in)	nc,
		integer	op,
		type(c_ptr), intent(inout)	strm
	)

inherited

Definition at line 97 of file gs_comm.f90.

◆ nbwait_vec() [2/2]

procedure, pass(this) gs_mpi::gs_mpi_t::nbwait_vec	(	class(gs_mpi_t), intent(inout)	this,
		real(kind=rp), dimension(nc*n), intent(inout)	u,
		integer, intent(in)	n,
		integer, intent(in)	nc,
		integer	op,
		type(c_ptr), intent(inout)	strm
	)

Definition at line 80 of file gs_mpi.f90.

◆ take_schedule()

procedure, pass(this) gs_comm::gs_comm_t::take_schedule	(	class(gs_comm_t), intent(inout)	this,
		class(gs_comm_t), intent(inout)	src
	)

inherited

Parameters

src	the backend to take the schedule from, left without one

Definition at line 90 of file gs_comm.f90.

Member Data Documentation

◆ ncompleted

integer gs_mpi::gs_mpi_t::ncompleted

Definition at line 64 of file gs_mpi.f90.

◆ recv_buf

real(kind=rp), dimension(:), allocatable gs_mpi::gs_mpi_t::recv_buf

Definition at line 53 of file gs_mpi.f90.

◆ recv_buf_v

real(kind=rp), dimension(:), allocatable gs_mpi::gs_mpi_t::recv_buf_v

Definition at line 69 of file gs_mpi.f90.

◆ recv_dof

type(stack_i4_t), dimension(:), allocatable gs_comm::gs_comm_t::recv_dof

inherited

Definition at line 58 of file gs_comm.f90.

◆ recv_indices

integer, dimension(:), allocatable gs_mpi::gs_mpi_t::recv_indices

Definition at line 62 of file gs_mpi.f90.

◆ recv_len

integer, dimension(:), allocatable gs_mpi::gs_mpi_t::recv_len

Definition at line 55 of file gs_mpi.f90.

◆ recv_offset

integer, dimension(:), allocatable gs_mpi::gs_mpi_t::recv_offset

Definition at line 57 of file gs_mpi.f90.

◆ recv_pe

integer, dimension(:), allocatable gs_comm::gs_comm_t::recv_pe

inherited

Definition at line 64 of file gs_comm.f90.

◆ recv_request

type(mpi_request), dimension(:), allocatable gs_mpi::gs_mpi_t::recv_request

Definition at line 59 of file gs_mpi.f90.

◆ recv_statuses

type(mpi_status), dimension(:), allocatable gs_mpi::gs_mpi_t::recv_statuses

Definition at line 63 of file gs_mpi.f90.

◆ send_buf

real(kind=rp), dimension(:), allocatable gs_mpi::gs_mpi_t::send_buf

Definition at line 51 of file gs_mpi.f90.

◆ send_buf_v

real(kind=rp), dimension(:), allocatable gs_mpi::gs_mpi_t::send_buf_v

Definition at line 68 of file gs_mpi.f90.

◆ send_dof

type(stack_i4_t), dimension(:), allocatable gs_comm::gs_comm_t::send_dof

inherited

Definition at line 55 of file gs_comm.f90.

◆ send_len

integer, dimension(:), allocatable gs_mpi::gs_mpi_t::send_len

Definition at line 55 of file gs_mpi.f90.

◆ send_offset

integer, dimension(:), allocatable gs_mpi::gs_mpi_t::send_offset

Definition at line 57 of file gs_mpi.f90.

◆ send_pe

integer, dimension(:), allocatable gs_comm::gs_comm_t::send_pe

inherited

Note: : this will usually be fewer than the total number of ranks size(send_pe) <= pe_size

Definition at line 62 of file gs_comm.f90.

◆ send_request

type(mpi_request), dimension(:), allocatable gs_mpi::gs_mpi_t::send_request

Definition at line 59 of file gs_mpi.f90.

◆ vec_ready

logical gs_comm::gs_comm_t::vec_ready = .false.

inherited

Definition at line 79 of file gs_comm.f90.

◆ vec_supported

logical gs_comm::gs_comm_t::vec_supported = .false.

inherited

Definition at line 68 of file gs_comm.f90.

The documentation for this type was generated from the following file:

/home/runner/work/neko/neko/src/gs/gs_mpi.f90

Public Member Functions

Public Attributes

Detailed Description

Member Function/Subroutine Documentation

◆ free() [1/2]

◆ free() [2/2]

◆ free_dofs()

◆ free_order()

◆ init() [1/2]

◆ init() [2/2]

◆ init_dofs()

◆ init_order()

◆ init_schedule()

◆ init_vec() [1/2]

◆ init_vec() [2/2]

◆ nbrecv() [1/2]

◆ nbrecv() [2/2]

◆ nbrecv_vec() [1/2]

◆ nbrecv_vec() [2/2]

◆ nbsend() [1/2]

◆ nbsend() [2/2]

◆ nbsend_vec() [1/2]

◆ nbsend_vec() [2/2]

◆ nbwait() [1/2]

◆ nbwait() [2/2]

◆ nbwait_vec() [1/2]

◆ nbwait_vec() [2/2]

◆ take_schedule()

Member Data Documentation

◆ ncompleted

◆ recv_buf

◆ recv_buf_v

◆ recv_dof

◆ recv_indices

◆ recv_len

◆ recv_offset

◆ recv_pe

◆ recv_request

◆ recv_statuses

◆ send_buf

◆ send_buf_v

◆ send_dof

◆ send_len

◆ send_offset

◆ send_pe

◆ send_request

◆ vec_ready

◆ vec_supported