Gather-scatter.

Data Types
type	gs_t
	Gather-scatter kernel. More...

Functions/Subroutines
subroutine	gs_init (gs, dofmap, bcknd, comm_bcknd)
	The runtime autotuning of the comm. backend, implemented in the gs_tune submodule: everything the selection needs (the candidate list, the NEKO_GS_TUNE parsing, switching a live gs over to another backend) is private to it, and only what gs_init drives is declared here.

subroutine, public	gs_comm_alloc (comm, comm_bcknd)
	Allocate a gather-scatter comm. backend of type comm_bcknd.

character(len=12) function, public	gs_comm_name (comm_bcknd)
	Name of the gather-scatter comm. backend comm_bcknd, right-adjusted for the log.

logical function, public	gs_comm_on_device (comm_bcknd)
	Whether the comm. backend comm_bcknd exchanges the shared dofs straight out of device memory rather than out of the host mirror of the shared buffer, which is what decides where the gather-scatter backend leaves them (gs_bcknd_tshared_on_host). A host backend on a device build is not device-resident: it drives the exchange from the mirror, at the price of a copy in each direction.

subroutine	gs_free (gs)
	Deallocate a gather-scatter kernel.

subroutine	gs_vec_alloc (gs)
	Allocate the buffers the fused vector gather-scatter needs, on the first gs_op_r3 that reaches the fused path.

subroutine	gs_init_mapping (gs)
	Setup mapping of dofs to gather-scatter operations.

subroutine	gs_schedule (gs)
	Schedule shared gather-scatter operations.

subroutine	gs_sort_i8 (a, ind, n)
	Heap sort for 64-bit integer arrays, returning the permutation ind. Local helper for gs_schedule (the generic math sort has no i8 variant).

subroutine	gs_op_fld (gs, u, op, event)
	Gather-scatter operation on a field u with op op.

subroutine	gs_op_r4 (gs, u, n, op, event)
	Gather-scatter operation on a rank 4 array.

subroutine	gs_op_vector (gs, u, n, op, event)
	Gather-scatter operation on a vector u with op op.

subroutine	gs_op_r3 (gs, u1, u2, u3, n, op, event)
	Gather-scatter operation on a 3-component vector of rank-4 arrays (u1, u2, u3) with op op; see gs_op_vector3.

subroutine	gs_op_vector3 (gs, u1, u2, u3, n, op, event)
	Gather-scatter operation on a 3-component vector (u1, u2, u3) with op op. When the comm backend supports a fused vector halo exchange, the three components are communicated in a single round; otherwise this falls back to three independent scalar gs_op_vector calls (identical result, 3 rounds). The on-node gather/scatter is always done per component (scalar), so only the communication cost is reduced. On device backends the caller's event is recorded once, after the last component's scatter (or superseded by hard synchronisation on host-mirrored comms), so a single event sync covers all components.

subroutine	gs_op_r3_device (gs, u1, u2, u3, n, op, nc, lo, so, m, l, tid, scatter_event)
	Device path for the fused 3-component gs. The device backend's shared gather/scatter always operate on its internal shared buffer, so each component is staged between that buffer and its column of the compact vector buffer gsshared_gs_v: through the host mirror when the comm backend is a host backend (shared_on_host, e.g. OpenCL/Metal or a device build with host MPI), or with device-to-device copies when the comm is device-resident (device MPI/NCCL/NVSHMEM). Apart from the column staging the two modes are identical; the comm backends transparently pick the host array or its device pointer, as in gs_op_vector.

Variables
integer, parameter	gs_tune_ntrials = 100
	Number of timed (and untimed, warm-up) gather-scatter operations per candidate in the runtime autotuning of comm. backends and device synchronisation strategies.

integer, parameter	gs_tune_nwarmup = 2

Function/Subroutine Documentation

◆ gs_comm_alloc()

subroutine, public gather_scatter::gs_comm_alloc	(	class(gs_comm_t), intent(out), allocatable	comm,
		integer, intent(in)	comm_bcknd
	)

Parameters

comm	the allocated backend. Must not hold a live backend on entry: intent(out) releases it without running its free, which would leak whatever resources it holds (symmetric memory, coarrays, uTofu VCQs). Both callers free the previous backend themselves, see gs_free and gs_comm_switch (in the gs_tune submodule).
comm_bcknd	comm. backend to allocate, one of the GS_COMM_* constants

Definition at line 412 of file gather_scatter.f90.

Here is the caller graph for this function:

◆ gs_comm_name()

character(len=12) function, public gather_scatter::gs_comm_name ( integer, intent(in) comm_bcknd )

Parameters

comm_bcknd comm. backend to name, one of the GS_COMM_* constants

Definition at line 448 of file gather_scatter.f90.

Here is the caller graph for this function:

◆ gs_comm_on_device()

logical function, public gather_scatter::gs_comm_on_device ( integer, intent(in) comm_bcknd )

Parameters

comm_bcknd comm. backend to check, one of the GS_COMM_* constants

Returns: whether the backend is device-resident

Definition at line 490 of file gather_scatter.f90.

Here is the caller graph for this function:

◆ gs_free()

subroutine gather_scatter::gs_free ( class(gs_t), intent(inout) gs )

private

Definition at line 504 of file gather_scatter.f90.

◆ gs_init()

subroutine gather_scatter::gs_init	(	class(gs_t), intent(inout)	gs,
		type(dofmap_t), intent(inout), target	dofmap,
		integer, optional	bcknd,
		integer, optional	comm_bcknd
	)

private

Time ntrials gather-scatter operations on u, returning the average wall time (s) per operation on this rank Benchmark the candidate comm. backends and leave gs holding the fastest one Bind the non-blocking synchronisation strategy of the device MPI comm. backend held by gs, benchmarking the strategies unless NEKO_GS_STRTGY names one Initialize a gather-scatter kernel

Parameters

dofmap,global	numbering of points and connectivity to base gs on
bcknd,backend	for executing the gs_ops
comm_bcknd,backend	for excuting the communication with

Definition at line 187 of file gather_scatter.f90.

◆ gs_init_mapping()

subroutine gather_scatter::gs_init_mapping ( type(gs_t), intent(inout), target gs )

private

Note: this might be a bit overkill, but having many collisions makes the init take too long. This is really critical to performance of the init

Definition at line 635 of file gather_scatter.f90.

Here is the call graph for this function:

◆ gs_op_fld()

subroutine gather_scatter::gs_op_fld	(	class(gs_t), intent(inout)	gs,
		type(field_t), intent(inout)	u,
		integer	op,
		type(c_ptr), intent(inout), optional	event
	)

private

Definition at line 1704 of file gather_scatter.f90.

Here is the caller graph for this function:

◆ gs_op_r3()

subroutine gather_scatter::gs_op_r3	(	class(gs_t), intent(inout)	gs,
		real(kind=rp), dimension(:,:,:,:), intent(inout), contiguous	u1,
		real(kind=rp), dimension(:,:,:,:), intent(inout), contiguous	u2,
		real(kind=rp), dimension(:,:,:,:), intent(inout), contiguous	u3,
		integer, intent(in)	n,
		integer	op,
		type(c_ptr), intent(inout), optional	event
	)

private

Definition at line 1809 of file gather_scatter.f90.

Here is the caller graph for this function:

◆ gs_op_r3_device()

subroutine gather_scatter::gs_op_r3_device	(	class(gs_t), intent(inout)	gs,
		real(kind=rp), dimension(n), intent(inout)	u1,
		real(kind=rp), dimension(n), intent(inout)	u2,
		real(kind=rp), dimension(n), intent(inout)	u3,
		integer, intent(in)	n,
		integer, intent(in)	op,
		integer, intent(in)	nc,
		integer, intent(in)	lo,
		integer, intent(in)	so,
		integer, intent(in)	m,
		integer, intent(in)	l,
		integer, intent(in)	tid,
		type(c_ptr), intent(inout)	scatter_event
	)

private

Definition at line 1954 of file gather_scatter.f90.

Here is the call graph for this function:

◆ gs_op_r4()

subroutine gather_scatter::gs_op_r4	(	class(gs_t), intent(inout)	gs,
		real(kind=rp), dimension(:,:,:,:), intent(inout), contiguous	u,
		integer, intent(in)	n,
		integer	op,
		type(c_ptr), intent(inout), optional	event
	)

private

Definition at line 1720 of file gather_scatter.f90.

Here is the caller graph for this function:

◆ gs_op_vector()

subroutine gather_scatter::gs_op_vector	(	class(gs_t), intent(inout)	gs,
		real(kind=rp), dimension(n), intent(inout)	u,
		integer, intent(in)	n,
		integer	op,
		type(c_ptr), intent(inout), optional	event
	)

private

Definition at line 1736 of file gather_scatter.f90.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ gs_op_vector3()

subroutine gather_scatter::gs_op_vector3	(	class(gs_t), intent(inout)	gs,
		real(kind=rp), dimension(n), intent(inout)	u1,
		real(kind=rp), dimension(n), intent(inout)	u2,
		real(kind=rp), dimension(n), intent(inout)	u3,
		integer, intent(in)	n,
		integer	op,
		type(c_ptr), intent(inout), optional	event
	)

private

Definition at line 1833 of file gather_scatter.f90.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ gs_schedule()

subroutine gather_scatter::gs_schedule ( type(gs_t), intent(inout), target gs )

private

Discovers, for every shared dof, the set of ranks that also hold it, using a canonical-owner rendezvous routed through the crystal router instead of the previous O(P) shifted neighbour exchange. Each dof global id is hashed to an owner rank, mod(gid, P); the owner gathers all holders and reflects, to each holder, the other holders. Each holder then registers the dof for both send and receive with every peer. The per-peer dof lists are ordered by dof global id, which gives the same ordering on both ranks of a pair (the matching invariant the gather-scatter exchange relies on) without ever transmitting a single sequenced key list. Scales to >1e5 ranks without O(P) buffers or unexpected-message buffer exhaustion.

Definition at line 1511 of file gather_scatter.f90.

Here is the call graph for this function:

◆ gs_sort_i8()

subroutine gather_scatter::gs_sort_i8	(	integer(i8), dimension(n), intent(inout)	a,
		integer, dimension(n), intent(out)	ind,
		integer, intent(in)	n
	)

private

Definition at line 1651 of file gather_scatter.f90.

◆ gs_vec_alloc()

subroutine gather_scatter::gs_vec_alloc ( class(gs_t), intent(inout) gs )

private

Parameters

gs	gather-scatter kernel to complete

Note: Rank local, not collective: a rank with no shared dofs never reaches the fused path, so neither this nor comminit_vec may communicate. Backends whose vector buffers come out of an allocation the whole run agrees on make them in init instead, see gs_comm_t.

Definition at line 616 of file gather_scatter.f90.

Variable Documentation

◆ gs_tune_ntrials

integer, parameter gather_scatter::gs_tune_ntrials = 100

Definition at line 146 of file gather_scatter.f90.

◆ gs_tune_nwarmup

integer, parameter gather_scatter::gs_tune_nwarmup = 2

private

Definition at line 147 of file gather_scatter.f90.

Data Types

Functions/Subroutines

Variables

Function/Subroutine Documentation

◆ gs_comm_alloc()

◆ gs_comm_name()

◆ gs_comm_on_device()

◆ gs_free()

◆ gs_init()

◆ gs_init_mapping()

◆ gs_op_fld()

◆ gs_op_r3()

◆ gs_op_r3_device()

◆ gs_op_r4()

◆ gs_op_vector()

◆ gs_op_vector3()

◆ gs_schedule()

◆ gs_sort_i8()

◆ gs_vec_alloc()

Variable Documentation

◆ gs_tune_ntrials

◆ gs_tune_nwarmup