Domain decomposition: `sdecomp`¶

APIs to achieve the domain decomposition are listed in this page.

Constructor¶

`construct`¶

Creating a structure sdecomp_info_t which contains all essential information to describe a decomposed domain and returns a pointer to it.
include/sdecomp.h¶
int (* const construct)(
    const MPI_Comm comm_default,
    const size_t ndims,
    const size_t * dims,
    const bool * periods,
    sdecomp_info_t ** info // out
);
Details
const MPI_Comm comm_default

MPI communicator which includes all processes which participate in this decomposition.

In most cases this should be MPI_COMM_WORLD.

const size_t ndims

Number of spatial dimensions of the domain you want to decompose. This should be 2 or 3 for two- / three-dimensional domains, respectively.
const size_t *dims

Number of processes in each dimension. Obviously the size (length) of this variable should be equivalent to the number of dimensions ndims, i.e.
#define NDIMS 2
size_t dims[NDIMS] = {0};
or
#define NDIMS 3
size_t dims[NDIMS] = {0};
You can specify how the domain is decomposed by the multiple processes using this parameter.

When you do not care the number of processes in each dimension:

Initialise all with zero, which asks MPI_Dims_create to automatically choose the number of processes in each dimension.

When the number of processes in each dimension is important to you:

You can manually decide the number of processes in each dimension. However, you need to specify all values properly. First, it is apparent that the products of dims (e.g. dims[0] * dims[1] when ndims is 2) should be equal to the total number of processes calling this initialiser (i.e., number of processes which belong to the communicator comm_default).

Second, dims[0] should be 1, since we do not decompose the domain in \(x\) direction (remember that we are dealing with pencils, not with cubes). Thus, for two-dimensional domains, dims[2] = {1, nprocs} is the only possible choice. For three-dimensional domains, there are several answers in general, as long as dims[1] * dims[2] == nprocs is satisfied, which is up to you.
periods

Periodicities in each dimension. In the above example, the boundaries are periodic (1) in \(x\) and \(y\) directions, while \(z\) direction is not (0), e.g. wall-bounded.
Example: automatic decomposition of a three-dimensional domain where y and z directions are periodic:
#define NDIMS 3
const size_t dims[NDIMS] = {0, 0, 0};
const bool periods[NDIMS] = {false, true, true};

sdecomp_info_t *info = NULL;
const int retval = sdecomp.construct(
    MPI_COMM_WORLD,
    NDIMS,
    dims,
    periods,
    &info
);
if(0 != retval){
   // failed to create info, error handling
}
Example: decomposition of a three-dimensional domain with 16384 processes, where both y and z directions are split into 128 pencils:
#define NDIMS 3
const size_t dims[NDIMS] = {1, 128, 128};
const bool periods[NDIMS] = {false, true, true};

// check total number of processes in MPI_COMM_WORLD,
//   which should be equal to dims[0] * dims[1] * dims[2]
int nprocs = 0;
MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
assert(dims[0] * dims[1] * dims[2] == nprocs);

sdecomp_info_t *info = NULL;
const int retval = sdecomp.construct(
    MPI_COMM_WORLD,
    NDIMS,
    dims,
    periods,
    &info
);
if(0 != retval){
   // failed to create info, error handling
}

Destructor¶

`destruct`¶

Destructor of sdecomp_info_t.
include/sdecomp.h¶
int (* const destruct)(
    sdecomp_info_t * sdecomp
);
Details
Example:
sdecomp.destruct(
    info
);
Here info is a pointer to sdecomp_info_t which was returned by sdecomp.construct.

Getter¶

`get_ndims`¶

Get the number of dimensions currently considered.
include/sdecomp.h¶
int (* const get_ndims)(
    const sdecomp_info_t * info,
    size_t * ndims // out
);
Details
This function retrieves ndims which was passed to sdecomp.construct when creating sdecomp_info_t *info.

Example:
size_t ndims = 0;
sdecomp.get_ndims(
    info,
    &ndims
);
2 or 3 should be assigned to ndims in this case, depending on your initialisation.

`get_comm_size`¶

Get the total number of processes in the default communicator.
include/sdecomp.h¶
int (* const get_comm_size)(
    const sdecomp_info_t * info,
    int * nprocs // out
);
Details
This function calls MPI_Comm_size, which returns the number of processes belong to the default communicator comm_default (an argument of sdecomp.construct).

Example:
int nprocs = 0;
sdecomp.get_comm_size(
    info,
    &nprocs
);

`get_comm_rank`¶

Get the ID of my rank in the default communicator.
include/sdecomp.h¶
int (* const get_comm_rank)(
    const sdecomp_info_t * info,
    int * myrank // out
);
Details
This function calls MPI_Comm_rank, which returns the rank of the calling process in the default communicator comm_default (an argument of sdecomp.construct).

Example:
int myrank = 0;
sdecomp.get_comm_rank(
    info,
    &myrank
);

`get_nprocs`¶

Get number of processes (number of pencils) in one direction.
include/sdecomp.h¶
int (* const get_nprocs)(
    const sdecomp_info_t * info,
    const sdecomp_pencil_t pencil,
    const sdecomp_dir_t dir,
    int * nprocs // out
);
Details
Example: get number of y1pencils (number of processes) in x direction:
const int ny1pencils_x = 0;
sdecomp.get_nprocs(
    info,
    SDECOMP_Y1PENCIL, // y1 pencil
    SDECOMP_XDIR,     // x direction
    &ny1pencils_x
);
In the case of the image, 3 is assigned to ny1pencils_x (see three pencils exist in x direction).

`get_myrank`¶

Get the position of my pencil.
include/sdecomp.h¶
int (* const get_myrank)(
    const sdecomp_info_t * info,
    const sdecomp_pencil_t pencil,
    const sdecomp_dir_t dir,
    int * myrank // out
);
Details
Example: get my location as a y1pencil in x direction:
const int myposition_1pencils_x = 0;
sdecomp.get_myrank(
    info,
    SDECOMP_Y1PENCIL, // y1 pencil
    SDECOMP_XDIR,     // x direction
    &myposition_1pencils_x
);
Let us imagine that this function is called by the green process. Then 1 is assigned to myposition_1pencils_x, since the green process is in the second position (notice that the index in C starts from 0).

`get_neighbours`¶

Get ranks of my neighbours under the communicator get_comm_cart gives.
include/sdecomp.h¶
int (* const get_neighbours)(
    const sdecomp_info_t * info,
    const sdecomp_pencil_t pencil,
    const sdecomp_dir_t dir,
    int neighbours[2] // out
);
Details
Example: get neighbour ranks of y1pencil in x direction:
const int neighbour_ranks[2] = {
   MPI_PROC_NULL,
   MPI_PROC_NULL
};
sdecomp.get_neighbours(
    info,
    SDECOMP_Y1PENCIL, // y1 pencil
    SDECOMP_XDIR,     // x direction
    neighbour_ranks
);
Let us imagine that this function is called by the green process. Then the ranks of the reddish and bluish processes are assigned to neighbour_ranks[0] and neighbour_ranks[1], respectively.

Note

MPI_PROC_NULL is assigned to neighbour_ranks if there is no neighbour there. This is the case when the constructor sdecomp.construct is called with the no-periodic condition in the direction.

Also my rank is assigned when there is only one process in the direction and the direction is periodic.

`get_pencil_mysize`¶

Get the size of my pencil.
include/sdecomp.h¶
int (* const get_pencil_mysize)(
    const sdecomp_info_t * info,
    const sdecomp_pencil_t pencil,
    const sdecomp_dir_t dir,
    const size_t glsize,
    size_t * mysize // out
);
Details
Example: get number of points I am responsible for as a y1pencil when the total number of points in x direction is 24:
const size_t glsize_x = 24;
size_t mysize_x = 0;
sdecomp.get_pencil_mysize(
    info,
    SDECOMP_Y1PENCIL, // y1 pencil
    SDECOMP_XDIR,     // x direction
    glsize_x,
    &mysize_x
);
Let us imagine that this function is called by the green process. Then 8 is assigned to mysize_x (this process is responsible for 8 grids in x direction).

`get_pencil_offset`¶

Get the offset of my pencil.
include/sdecomp.h¶
int (* const get_pencil_offset)(
    const sdecomp_info_t * info,
    const sdecomp_pencil_t pencil,
    const sdecomp_dir_t dir,
    const size_t glsize,
    size_t * offset // out
);
Details
Example: get number of points up to my position as a y1pencil when the total number of points in x direction is 1024:
const size_t glsize = 1024;
size_t offset = 0;
sdecomp.get_pencil_offset(
    info,
    SDECOMP_Y1PENCIL,
    SDECOMP_XDIR,
    glsize_x,
    &offset_x
);
Let us imagine that this function is called by the green process. Then 8 is assigned to offset_x, since the reddish process (having 8 grids inside) sits in front of the green one.

`get_comm_cart`¶

Get the communicator holding the Cartesian topology of x1pencil.
include/sdecomp.h¶
int (* const get_comm_cart)(
    const sdecomp_info_t * info,
    MPI_Comm * comm // out
);
Details
This is useful to call MPI_Cart_xxx directly, or to perform advanced operations.

Example:
MPI_Comm comm_cart = MPI_COMM_NULL;
sdecomp.get_comm_cart(
    info,
    &comm_cart
);

Domain decomposition: `sdecomp`¶

Constructor¶

`construct`¶

Destructor¶

`destruct`¶

Getter¶

`get_ndims`¶

`get_comm_size`¶

`get_comm_rank`¶

`get_nprocs`¶

`get_myrank`¶

`get_neighbours`¶

`get_pencil_mysize`¶

`get_pencil_offset`¶

`get_comm_cart`¶

SimpleDecomp

Navigation

Related Topics

Domain decomposition: sdecomp¶

Constructor¶

construct¶

Destructor¶

destruct¶

Getter¶

get_ndims¶

get_comm_size¶

get_comm_rank¶

get_nprocs¶

get_myrank¶

get_neighbours¶

get_pencil_mysize¶

get_pencil_offset¶

get_comm_cart¶

Domain decomposition: `sdecomp`¶

`construct`¶

`destruct`¶

`get_ndims`¶

`get_comm_size`¶

`get_comm_rank`¶

`get_nprocs`¶

`get_myrank`¶

`get_neighbours`¶

`get_pencil_mysize`¶

`get_pencil_offset`¶

`get_comm_cart`¶