For MPI development and runtime support, \OHPC{} provides pre-packaged builds
for a variety of MPI families and transport layers. Currently available options
and their applicability to various network transports are summarized in
Table~\ref{table:mpi}.  The command that follows installs a starting set of MPI
families that are compatible with both ethernet and high-speed fabrics.

\iftoggleverb{isx86}
% x86_64

\begin{table}[h]
\caption{Available MPI variants} \label{table:mpi}
\centering
\begin{tabular}{@{\hspace*{0.2cm}} *5l @{}}    \toprule
                                  & Ethernet (TCP)                 & \InfiniBand{}                  & \IntelR{} Omni-Path            \\ \midrule
\rowcolor{black!10} MPICH (ofi) & \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} \\
 MPICH (ucx)       & \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} \\
\rowcolor{black!10} MVAPICH2                          &                                & \multicolumn{1}{c}{\checkmark} &                                \\
MVAPICH2 (psm2) &                              &                                & \multicolumn{1}{c}{\checkmark} \\
\rowcolor{black!10} OpenMPI (ofi/ucx)            & \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} \\
%\rowcolor{black!10} OpenMPI (PMIx) & \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} \\ \bottomrule
\end{tabular}
\end{table}

\else
% aarch64

\begin{table}[h]
\caption{Available MPI builds} \label{table:mpi}
\centering
\begin{tabular}{@{\hspace*{0.2cm}} *5l @{}}    \toprule
                                  & Ethernet (TCP)                 & \InfiniBand{}                              \\ \midrule
\rowcolor{black!10} MPICH         & \multicolumn{1}{c}{\checkmark} &                                            \\
\rowcolor{black!10} OpenMPI                           & \multicolumn{1}{c}{\checkmark} & \multicolumn{1}{c}{\checkmark} \\
\end{tabular}
\end{table}

\fi


% begin_ohpc_run
% ohpc_comment_header Install MPI Stacks \ref{sec:mpi}
% ohpc_command if [[ ${enable_mpi_defaults} -eq 1 ]];then
% ohpc_indent 5
\begin{lstlisting}[language=bash]
[sms](*\#*) (*\install*) openmpi5-gnu15-ohpc mpich-ofi-gnu15-ohpc
\end{lstlisting}
% ohpc_indent 0
% ohpc_command fi
% end_ohpc_run

Note that OpenHPC 2.x introduces the use of two related transport layers for
the MPICH and OpenMPI builds that support a variety of underlying
fabrics: \href{https://www.openucx.org}{UCX} (Unified Communication X)
and \href{https://ofiwg.github.io/libfabric/}{OFI} (OpenFabrics interfaces).
In the case of OpenMPI, a monolithic build is provided which supports both
transports and end-users can customize their runtime preferences with
environment variables. For MPICH, two separate builds are provided and the
example above highlighted installing the {\texttt ofi} variant.  However, the
packaging is designed such that both versions can be installed simultaneously
and users can switch between the two via normal module command
semantics. Alternatively, a site can choose to install the {\texttt ucx} variant
instead as a drop-in MPICH replacement:

\begin{lstlisting}[language=bash]
[sms](*\#*) (*\install*) mpich-ucx-gnu15-ohpc
\end{lstlisting}

In the case where both MPICH variants are installed, two modules will be
visible in the end-user environment and an example of this configuration is
highlighted is below.

\begin{lstlisting}[language=bash]
[sms](*\#*) module avail mpich

-------------------- /opt/ohpc/pub/moduledeps/gnu15---------------------
   mpich/3.4.3-ofi    mpich/3.4.3-ucx (D)
\end{lstlisting}

If your system includes \InfiniBand{} and you enabled underlying support in
\S\ref{sec:add_ofed} and \S\ref{sec:addl_customizations}, an additional
MVAPICH2 family is available for use:

% begin_ohpc_run
% ohpc_validation_newline
% ohpc_command if [[ ${enable_ib} -eq 1 ]];then
% ohpc_indent 5
\begin{lstlisting}[language=bash]
[sms](*\#*) (*\install*) mvapich2-gnu15-ohpc
\end{lstlisting}
% ohpc_indent 0
% ohpc_command fi
% end_ohpc_run

Alternatively, if your system includes \IntelR{} \OmniPath{}, use the (\texttt{psm2})
variant of MVAPICH2 instead:

% begin_ohpc_run
% ohpc_command if [[ ${enable_opa} -eq 1 ]];then
% ohpc_indent 5
\begin{lstlisting}[language=bash]
[sms](*\#*) (*\install*) mvapich2-psm2-gnu15-ohpc
\end{lstlisting}
% ohpc_indent 0
% ohpc_command fi
% end_ohpc_run

%%--
%% https://github.com/openhpc/ohpc/issues/1273
%% disabling until we can get pmix/openmpi/slurm to play nicely
%%--
%% An additional OpenMPI build variant is listed in Table~\ref{table:mpi} which
%% enables \href{https://pmix.github.io/pmix/}{\color{blue}{PMIx}} job launch
%% support for use with \SLURM{}. This optional variant is
%% available as \texttt{openmpi5-pmix-slurm-gnu9-ohpc}.
