MPI (Message Passing Interface) is a standard interface for high performance computing middleware. MPI uses typed messages and provides a variety of message exchange functions for point-to-point communication, collective communication functions designed for coordinated multiparty message exchange, optionally optimized for particular communication topologies, remote memory access functions, and additional functions for coordinated I/O. The standard defines the interface in a language agnostic notation, language bindings are defined for C and Fortran.
Initialization assumes either a world model or a session model. The world model assumes a flat application architecture where all processes interact. The session model assumes a modular application architecture where communication is restricted to processes executing individual modules. Cluster resources in session model are identified using URI names configured by cluster administrator.
World Model. For programs where all processes coordinate together
int MPI_Init (int *argc, char ***argv); int MPI_Init_thread (int *argc, char ***argv, int required, int *provided); int MPI_Query_thread (int *provided); int MPI_Is_thread_main (int *flag); int MPI_Initialized (int *flag); int MPI_Finalize (void);
single thread in this process
multiple threads but only main thread calls MPI
multiple threads but only one MPI call at a time
multiple threads and multiple MPI calls at a time
Session Model. For programs where processes coordinate within components
int MPI_Session_init (MPI_Info info, MPI_Errhandler errhandler, MPI_Session *session); int MPI_Session_get_num_psets (MPI_Session session, MPI_Info info, int *npset_names); int MPI_Session_get_nth_pset (MPI_Session session, MPI_Info info, int n, int *pset_len, char *pset_name); int MPI_Session_finalize (MPI_Session *session);
Process sets are administratively defined groups of processes
arbitrary overlap possible
intended to express shared resource scopes
mpi://SELF and mpi://WORLD always exist
Dynamic Process Model. For explicit control over process lifecycle
int MPI_Comm_spawn (
const char *command, char *argv[], int maxprocs, MPI_Info info,
int root, MPI_Comm comm, MPI_Comm *intercomm,
int array_of_errcodes []);
int MPI_Comm_spawn_multiple (
int count, char *array_of_commands [], char **array_of_argv [],
const int array_of_maxprocs [], const MPI_Info array_of_info [],
int root, MPI_Comm comm, MPI_Comm *intercomm,
int array_of_errcodes [])
int MPI_Comm_get_parent (MPI_Comm *parent);
children have separate MPI_COMM_WORLD
Configuration. A set of key value pairs used to provide additional configuration
int MPI_Info_create (MPI_Info *info); int MPI_Info_free (MPI_Info *info); int MPI_Info_dup (MPI_Info info, MPI_Info *newinfo); int MPI_Info_set (MPI_Info info, const char *key, const char *value); int MPI_Info_get_nkeys (MPI_Info info, int *nkeys); int MPI_Info_get_nthkey (MPI_Info info, int n, char *key); int MPI_Info_get_string (MPI_Info info, const char *key, int *buflen, char *value, int *flag); int MPI_Info_delete (MPI_Info info, const char *key);
Groups. A group contains processes belonging to an application
MPI_GROUP_EMPTY
int MPI_Group_size (MPI_Group group, int *size);
int MPI_Group_rank (MPI_Group group, int *rank);
int MPI_Group_translate_ranks (
MPI_Group group1, int n, const int ranks1 [],
MPI_Group group2, int ranks2 []);
int MPI_Group_compare (MPI_Group group1, MPI_Group group2, int *result);
int MPI_Group_union (MPI_Group group1, MPI_Group group2, MPI_Group *newgroup);
int MPI_Group_difference (MPI_Group group1, MPI_Group group2, MPI_Group *newgroup);
int MPI_Group_intersection (MPI_Group group1, MPI_Group group2, MPI_Group *newgroup);
int MPI_Group_incl (MPI_Group group, int n, const int ranks [], MPI_Group *newgroup);
int MPI_Group_excl (MPI_Group group, int n, const int ranks [], MPI_Group *newgroup);
int MPI_Group_range_incl (MPI_Group group, int n, int ranges [][3], MPI_Group *newgroup);
int MPI_Group_range_excl (MPI_Group group, int n, int ranges [][3], MPI_Group *newgroup);
int MPI_Comm_group (MPI_Comm comm, MPI_Group *group);
int MPI_Group_from_session_pset (MPI_Session session, const char *pset_name, MPI_Group *newgroup);
int MPI_Group_free (MPI_Group *group);
created from other groups in the world model
created from process sets in the session model
processes addressed using rank from 0 to size - 1
Communicators. A communicator represents one or two groups in communication context
MPI_COMM_SELF
MPI_COMM_WORLD
int MPI_Comm_size (MPI_Comm comm, int *size);
int MPI_Comm_rank (MPI_Comm comm, int *rank);
int MPI_Comm_compare (MPI_Comm comm1, MPI_Comm comm2, int *result);
int MPI_Comm_dup (MPI_Comm comm, MPI_Comm *newcomm);
int MPI_Comm_dup_with_info (MPI_Comm comm, MPI_Info info, MPI_Comm *newcomm);
int MPI_Comm_create (MPI_Comm comm, MPI_Group group, MPI_Comm *newcomm);
int MPI_Comm_create_group (MPI_Comm comm, MPI_Group group, int tag, MPI_Comm *newcomm);
int MPI_Comm_create_from_group (MPI_Group group, const char *stringtag, MPI_Info info, MPI_Errhandler errhandler, MPI_Comm *newcomm);
int MPI_Intercomm_create (
MPI_Comm local_comm, int local_leader,
MPI_Comm peer_comm, int remote_leader,
int tag, MPI_Comm *newintercomm);
int MPI_Intercomm_create_from_groups (
MPI_Group local_group, int local_leader,
MPI_Group remote_group, int remote_leader,
const char *stringtag, MPI_Info info, MPI_Errhandler errhandler, MPI_Comm *newintercomm);
int MPI_Comm_split (MPI_Comm comm, int color, int key, MPI_Comm *newcomm);
int MPI_Comm_split_type (MPI_Comm comm, int split_type, int key, MPI_Info info, MPI_Comm *newcomm);
int MPI_Intercomm_merge (MPI_Comm intercomm, int high, MPI_Comm *newintracomm);
int MPI_Comm_free (MPI_Comm *comm);
int MPI_Comm_set_info (MPI_Comm comm, MPI_Info info);
int MPI_Comm_get_info (MPI_Comm comm, MPI_Info *info_used);
int MPI_Comm_test_inter (MPI_Comm comm, int *flag);
int MPI_Comm_remote_size (MPI_Comm comm, int *size);
int MPI_Comm_remote_group (MPI_Comm comm, MPI_Group *group);
communication within single group
communication between two groups
int MPI_Send (
const void *buf, int count, MPI_Datatype datatype,
int dest, int tag, MPI_Comm comm);
int MPI_Send_c (
const void *buf, MPI_Count count, MPI_Datatype datatype,
int dest, int tag, MPI_Comm comm);
address of send buffer
number of elements in send buffer
datatype of each send buffer element
rank of destination
message tag
communicator
int MPI_Recv (
void *buf, int count, MPI_Datatype datatype,
int source, int tag, MPI_Comm comm,
MPI_Status *status);
int MPI_Recv_c (
void *buf, MPI_Count count, MPI_Datatype datatype,
int source, int tag, MPI_Comm comm,
MPI_Status *status);
address of receive buffer
maximum number of elements in receive buffer
datatype of each receive buffer element
rank of source or MPI_ANY_SOURCE
message tag or MPI_ANY_TAG
communicator
status object
int MPI_Get_count (const MPI_Status *status, MPI_Datatype datatype, int *count); int MPI_Get_count_c (const MPI_Status *status, MPI_Datatype datatype, MPI_Count *count);
int MPI_Sendrecv (
const void *sendbuf, int sendcount, MPI_Datatype sendtype,
int dest, int sendtag,
void *recvbuf, int recvcount, MPI_Datatype recvtype,
int source, int recvtag, MPI_Comm comm,
MPI_Status *status);
int MPI_Sendrecv_c (
const void *sendbuf, MPI_Count sendcount, MPI_Datatype sendtype,
int dest, int sendtag,
void *recvbuf, MPI_Count recvcount, MPI_Datatype recvtype,
int source, int recvtag, MPI_Comm comm,
MPI_Status *status);
int MPI_Send (const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm); int MPI_Bsend (const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm); int MPI_Ssend (const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm); int MPI_Rsend (const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm); int MPI_Buffer_attach (void *buffer, int size); int MPI_Buffer_attach_c (void *buffer, MPI_Count size); int MPI_Buffer_detach (void *buffer_addr, int *size); int MPI_Buffer_detach_c (void *buffer_addr, MPI_Count *size);
may block, buffer available on return, asynchronous
does not block, uses supplied buffers, buffer available on return, asynchronous
may block, target must be receiving otherwise function fails
may block, target must be receiving otherwise undefined
int MPI_Isend (
const void *buf, int count, MPI_Datatype datatype,
int dest, int tag, MPI_Comm comm,
MPI_Request *request);
int MPI_Isend_c (
const void *buf, MPI_Count count, MPI_Datatype datatype,
int dest, int tag, MPI_Comm comm,
MPI_Request *request);
int MPI_Ibsend (...);
int MPI_Issend (...);
int MPI_Irsend (...);
int MPI_Irecv (
void *buf, int count, MPI_Datatype datatype,
int source, int tag, MPI_Comm comm,
MPI_Request *request);
int MPI_Irecv_c (
void *buf, MPI_Count count, MPI_Datatype datatype,
int source, int tag, MPI_Comm comm,
MPI_Request *request);
int MPI_Iprobe (int source, int tag, MPI_Comm comm, int *flag, MPI_Status *status);
int MPI_Improbe (int source, int tag, MPI_Comm comm, int *flag, MPI_Message *message, MPI_Status *status);
int MPI_Imrecv (void *buf, int count, MPI_Datatype datatype, MPI_Message *message, MPI_Request *request);
int MPI_Imrecv_c (void *buf, MPI_Count count, MPI_Datatype datatype, MPI_Message *message, MPI_Request *request);
int MPI_Wait (MPI_Request *request, MPI_Status *status);
int MPI_Waitany (int count, MPI_Request array_of_requests [], int *index, MPI_Status *status);
int MPI_Waitall (int count, MPI_Request array_of_requests [], MPI_Status array_of_statuses []);
int MPI_Waitsome (int incount, MPI_Request array_of_requests [], int *outcount, int array_of_indices [], MPI_Status array_of_statuses []);
int MPI_Test (MPI_Request *request, int *flag, MPI_Status *status);
int MPI_Testany (int count, MPI_Request array_of_requests [], int *index, int *flag, MPI_Status *status);
int MPI_Testall (int count, MPI_Request array_of_requests [], int *flag, MPI_Status array_of_statuses []);
int MPI_Testsome (int incount, MPI_Request array_of_requests [], int *outcount, int array_of_indices [], MPI_Status array_of_statuses []);
int MPI_Request_free (MPI_Request *request);
int MPI_Request_get_status (MPI_Request request, int *flag, MPI_Status *status);
int MPI_Cancel (MPI_Request *request);
// Persistent communication requests serve to efficiently initiate repetitive communication.
int MPI_Send_init_c (
const void *buf, MPI_Count count, MPI_Datatype datatype,
int dest, int tag, MPI_Comm comm,
MPI_Request *request);
int MPI_Recv_init_c (
void *buf, MPI_Count count, MPI_Datatype datatype,
int source, int tag, MPI_Comm comm,
MPI_Request *request);
int MPI_Start (MPI_Request *request);
int MPI_Startall (int count, MPI_Request array_of_requests []);
// Partitioned communication requests deliver message content in independent partitions.
int MPI_Psend_init (
const void *buf, int partitions, MPI_Count count, MPI_Datatype datatype,
int dest, int tag, MPI_Comm comm, MPI_Info info,
MPI_Request *request);
int MPI_Precv_init (
void *buf, int partitions, MPI_Count count, MPI_Datatype datatype,
int source, int tag, MPI_Comm comm, MPI_Info info,
MPI_Request *request);
int MPI_Start (MPI_Request *request);
int MPI_Pready (int partition, MPI_Request request);
int MPI_Pready_range (int partition_low, int partition_high, MPI_Request request);
int MPI_Pready_list (int length, const int array_of_partitions [], MPI_Request request);
int MPI_Wait (...);
Selected Basic Types.
signed integer types
unsigned integer types
floating point types
character data types
exact size signed integer types
exact size unsigned integer types
buffer with raw data
buffer with packed data
Local Derived Types.
int MPI_Type_commit (MPI_Datatype *datatype);
int MPI_Type_free (MPI_Datatype *datatype);
int MPI_Type_dup (MPI_Datatype oldtype, MPI_Datatype *newtype);
int MPI_Type_contiguous (int count, MPI_Datatype oldtype, MPI_Datatype *newtype);
int MPI_Type_vector (int count, int blocklength, int stride, MPI_Datatype oldtype, MPI_Datatype *newtype);
int MPI_Type_indexed (
int count,
const int array_of_blocklengths [],
const int array_of_displacements [],
MPI_Datatype oldtype,
MPI_Datatype *newtype);
int MPI_Type_create_struct (
int count,
const int array_of_blocklengths [],
const MPI_Aint array_of_displacements [],
const MPI_Datatype array_of_types [],
MPI_Datatype *newtype);
int MPI_Type_create_subarray (
int ndims,
const int array_of_sizes [], const int array_of_subsizes [],
const int array_of_starts [],
int order,
MPI_Datatype oldtype,
MPI_Datatype *newtype);
elements of basic types
offset for each element
also versions with byte offsets
also functions for introspecting derived types
also functions for data import and export in canonical format
row major order
column major order
Distributed Derived Types.
int MPI_Type_create_darray (
int size, int rank,
int ndims,
const int array_of_gsizes [],
const int array_of_distribs [], const int array_of_dargs [],
const int array_of_psizes [],
int order,
MPI_Datatype oldtype,
MPI_Datatype *newtype);
sequential block distribution (AAABBBCCC...)
cyclic element distribution (ABC...ABC...ABC...)
Conversions.
MPI types provided by communication parties must be the same
representation conversion for portability is performed as necessary
representation of derived types with byte offsets may not be portable
int MPI_Gather (
const void *sendbuf, int sendcount, MPI_Datatype sendtype,
void *recvbuf, int recvcount, MPI_Datatype recvtype,
int root, MPI_Comm comm);
int MPI_Gather_c(
const void *sendbuf, MPI_Count sendcount, MPI_Datatype sendtype,
void *recvbuf, MPI_Count recvcount, MPI_Datatype recvtype,
int root, MPI_Comm comm);
int MPI_Gatherv (
const void *sendbuf, int sendcount, MPI_Datatype sendtype,
void *recvbuf, const int recvcounts [], const int displs [], MPI_Datatype recvtype,
int root, MPI_Comm comm);
int MPI_Gatherv_c (...);
int MPI_Igather (
const void *sendbuf, int sendcount, MPI_Datatype sendtype,
void *recvbuf, int recvcount, MPI_Datatype recvtype,
int root, MPI_Comm comm,
MPI_Request *request);
int MPI_Igather_c (...);
int MPI_Igatherv (
const void *sendbuf, int sendcount, MPI_Datatype sendtype,
void *recvbuf, const int recvcounts [], const int displs [],
MPI_Datatype recvtype, int root, MPI_Comm comm,
MPI_Request *request);
int MPI_Igatherv_c (...);
int MPI_Gather_init (
const void *sendbuf, int sendcount, MPI_Datatype sendtype,
void *recvbuf, int recvcount, MPI_Datatype recvtype,
int root, MPI_Comm comm, MPI_Info info,
MPI_Request *request);
int MPI_Gather_init_c (...);
int MPI_Gatherv_init (
const void *sendbuf, int sendcount, MPI_Datatype sendtype,
void *recvbuf, const int recvcounts [], const int displs [], MPI_Datatype recvtype,
int root, MPI_Comm comm, MPI_Info info,
MPI_Request *request);
int MPI_Gatherv_init_c (...);
sender A, receivers A, A, A
senders A, B, C, receiver ABC
sender ABC, receivers A, B, C
senders A, B, C, receivers ABC, ABC, ABC
senders ABC, DEF, GHI, receivers ADG, BEH, CFI
senders A, B, C, receiver A+B+C
senders A, B, C, receivers A+B+C, A+B+C, A+B+C
senders ABC, DEF, GHI, receivers A+D+G, B+E+H, C+F+I
senders A, B, C, receivers A, A+B, A+B+C
senders A, B, C, receivers N/A, A, A+B
rendez vous
semantics differ for intra and inter communication
intra communication can use special argument for single buffer
MPI_SUM, MPI_PROD MPI_MIN, MPI_MINLOC MPI_MAX, MPI_MAXLOC MPI_LAND, MPI_LOR, MPI_LXOR MPI_BAND, MPI_BOR, MPI_BXOR int MPI_Op_create (MPI_User_function *user_fn, int commute, MPI_Op *op); int MPI_Op_free (MPI_Op *op); typedef void MPI_User_function (void *invec, void *inoutvec, int *len, MPI_Datatype *datatype);
must be associative
may be commutative
int MPI_Cart_create (
MPI_Comm comm_old,
int ndims, const int dims [], const int periods [],
int reorder, MPI_Comm *comm_cart);
int MPI_Cart_sub (MPI_Comm comm, const int remain_dims [], MPI_Comm *newcomm);
int MPI_Graph_create (
MPI_Comm comm_old,
int nnodes, const int index [], const int edges [],
int reorder, MPI_Comm *comm_graph);
int MPI_Dist_graph_create_adjacent (
MPI_Comm comm_old,
int indegree, const int sources [], const int sourceweights [],
int outdegree, const int destinations [], const int destweights [],
MPI_Info info, int reorder, MPI_Comm *comm_dist_graph);
int MPI_Dist_graph_create (
MPI_Comm comm_old,
int n, const int sources [], const int degrees [], const int destinations [], const int weights [],
MPI_Info info, int reorder, MPI_Comm *comm_dist_graph);
cartesian grid with optionally periodic dimensions
centralized graph definition with node degrees and flattened edge list
distributed graph definition which specifies incoming and outgoing edges at each node
distributed graph definition which specifies arbitrary subset of edges at each node
int MPI_Cart_rank (MPI_Comm comm, const int coords [], int *rank);
int MPI_Cart_coords (MPI_Comm comm, int rank, int maxdims, int coords []);
int MPI_Cart_shift (MPI_Comm comm, int direction, int disp, int *rank_source, int *rank_dest);
int MPI_Graph_neighbors (MPI_Comm comm, int rank, int maxneighbors, int neighbors []);
int MPI_Dist_graph_neighbors (
MPI_Comm comm,
int maxindegree, int sources [], int sourceweights [],
int maxoutdegree, int destinations [], int destweights []);
int MPI_Neighbor_allgather (
const void *sendbuf, int sendcount, MPI_Datatype sendtype,
void *recvbuf, int recvcount, MPI_Datatype recvtype,
MPI_Comm comm);
int MPI_Neighbor_allgatherv (
const void *sendbuf, int sendcount, MPI_Datatype sendtype,
void *recvbuf, const int recvcounts [], const int displs [], MPI_Datatype recvtype,
MPI_Comm comm);
int MPI_Neighbor_alltoall (
const void *sendbuf, int sendcount, MPI_Datatype sendtype,
void *recvbuf, int recvcount, MPI_Datatype recvtype,
MPI_Comm comm);
int MPI_Neighbor_alltoallv (
const void *sendbuf, const int sendcounts [], const int sdispls [], MPI_Datatype sendtype,
void *recvbuf, const int recvcounts [], const int rdispls [], MPI_Datatype recvtype,
MPI_Comm comm);
int MPI_Neighbor_alltoallw (
const void *sendbuf, const int sendcounts [], const MPI_Aint sdispls [], const MPI_Datatype sendtypes [],
void *recvbuf, const int recvcounts [], const MPI_Aint rdispls [], const MPI_Datatype recvtypes [],
MPI_Comm comm);
int MPI_Win_create (
void *base, MPI_Aint size, int disp_unit,
MPI_Info info, MPI_Comm comm, MPI_Win *win);
int MPI_Win_create_c (
void *base, MPI_Aint size, MPI_Aint disp_unit,
MPI_Info info, MPI_Comm comm, MPI_Win *win);
int MPI_Win_allocate (
MPI_Aint size, int disp_unit,
MPI_Info info, MPI_Comm comm, void *baseptr, MPI_Win *win);
int MPI_Win_allocate_c (
MPI_Aint size, MPI_Aint disp_unit,
MPI_Info info, MPI_Comm comm, void *baseptr, MPI_Win *win);
int MPI_Win_allocate_shared (
MPI_Aint size, int disp_unit,
MPI_Info info, MPI_Comm comm, void *baseptr, MPI_Win *win);
int MPI_Win_allocate_shared_c (
MPI_Aint size, MPI_Aint disp_unit,
MPI_Info info, MPI_Comm comm, void *baseptr, MPI_Win *win);
int MPI_Win_create_dynamic (MPI_Info info, MPI_Comm comm, MPI_Win *win);
int MPI_Win_attach (MPI_Win win, void *base, MPI_Aint size);
int MPI_Win_detach (MPI_Win win, const void *base);
int MPI_Win_free (MPI_Win *win);
unit size used in scaling remote offsets
shared windows only possible when hardware architecture permits that
dynamic windows permit dynamic attaching and detaching of memory with given size
Remote memory access requires synchronization. Passive target synchronization assumes the target does not explicitly participate in synchronization and the origin process explicitly delimits access epochs with calls to MPI_Win_flush or MPI_Win_lock and MPI_Win_unlock. Active target synchronization assumes the target makes data available by explicitly delimiting exposure epochs with calls to MPI_Win_fence or MPI_Win_post and MPI_Win_wait, and the origin process explicitly delimits access epochs with calls to MPI_Win_fence or MPI_Win_start and MPI_Win_complete.
int MPI_Put (
void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
int target_rank, int target_disp, int target_count, MPI_Datatype target_datatype,
MPI_Win win);
int MPI_Get (
void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype,
MPI_Win win);
int MPI_Accumulate (
const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype,
MPI_Op op,
MPI_Win win);
int MPI_Get_accumulate (
const void *origin_addr, int origin_count, MPI_Datatype origin_datatype,
void *result_addr, int result_count, MPI_Datatype result_datatype,
int target_rank, MPI_Aint target_disp, int target_count, MPI_Datatype target_datatype,
MPI_Op op,
MPI_Win win);
int MPI_Fetch_and_op (
const void *origin_addr, void *result_addr, MPI_Datatype datatype,
int target_rank, MPI_Aint target_disp,
MPI_Op op,
MPI_Win win);
int MPI_Compare_and_swap (
const void *origin_addr, const void *compare_addr, void *result_addr, MPI_Datatype datatype,
int target_rank, MPI_Aint target_disp,
MPI_Win win);
local access requires explicit synchronization
Passive Target Synchronization.
int MPI_Win_flush (int rank, MPI_Win win); int MPI_Win_flush_all (MPI_Win win); int MPI_Win_flush_local (int rank, MPI_Win win); int MPI_Win_flush_local_all (MPI_Win win); int MPI_Win_lock (int lock_type, int rank, int assert, MPI_Win win); int MPI_Win_lock_all (int assert, MPI_Win win); int MPI_Win_unlock (int rank, MPI_Win win); int MPI_Win_unlock_all (MPI_Win win);
Active Target Synchronization.
int MPI_Win_fence (int assert, MPI_Win win); int MPI_Win_start (MPI_Group group, int assert, MPI_Win win); int MPI_Win_complete (MPI_Win win); int MPI_Win_post (MPI_Group group, int assert, MPI_Win win); int MPI_Win_wait (MPI_Win win);
assert parameter specifies application hints for optimization
starting and completing delineates epoch of remote window access
posting and waiting delineates epoch of exposure to remote window access
The MPI Forum Home Page. https://www.mpi-forum.org
The MPICH Project Home Page. https://www.mpich.org
The OpenMPI Project Home Page. https://www.open-mpi.org