Scientific Computing & Visualization | Computation | Katana Cluster | How to Select MPI Implementation and Compilers

This Site

How to Select MPI Implementation and Compilers

On the Katana, two groups of compilers, from PGI and GNU, are available to users. In addition, two different MPI implementations, namely openmpi and mpich, are also installed. The system defaults the MPI implementation to openmpi and defaults the compilers to those of PGI. Based on limited testing, we have found that both MPI implementations and both compiler groups perform comparably. We recommend that the system-default setting be used for general applications to avoid confusions. However, users may change the MPI implementation to mpich and the compilers to gnu for specific applications. For example, if you want to examine the performance of your code using the PGI profiler pgprof, the MPI implementation must be set to mpich. On the other hand, if your code uses MPI-2 (MPI-1 extension) functionalities such as MPI_Put and MPI_Get, they are supported in the (default) "openmpi" but not in the "mpich" MPI implementation.

Caution: Due to these choices, it is possible to compile codes under one set of conditions and, unknowingly, attempting to run jobs (through mpirun) under a different set of conditions. Therefore, care must be exercised to ensure consistency between code compilation and running jobs if you need to use the non-default system settings.

MPI_IMPLEMENTATION -- choices for this environment variable are:
openmpi (default) and mpich
To change the default:
```
katana:~ % setenv MPI_IMPLEMENTATION mpich
```
Note: This will affect the MPI run script mpirun as well.
MPI_COMPILER -- choices for this environment variable are:
pgi (default) and gnu
To change the default:
```
katana:~ % setenv MPI_COMPILER gnu
```

Given the two options each for MPI implementation and compilers, there are 4 distinct possibilities for what mpicc ( and mpif90, ... ) represents. If the user takes no action, mpicc uses pgcc along with the openmpi MPI libraries to compile programs.


katana:~ % mpif77 -o myexec myprogram.f

katana:~ % mpif90 -o myexec myprogram.f90

katana:~ % mpicc -o myexec myprogram.c

MPI manpages

The normal "man" command will return the MPI manpages associated with the openmpi implementation of MPI.

The "mpiman" command will return MPI manpages associated with the implementation specified by the "MPI_IMPLEMENTATION" variable.


katana:~ % mpiman mpicc

lists the mpicc manpage under openmpi.


katan:~ % setenv MPI_IMPLEMENTATION mpich
katana:~ % mpiman mpirun

lists the manpage of mpirun under mpich MPI implementation.

Miscellaneous

Is there a reason or circumstance for me not to use the system-default settings for MPI_IMPLEMENTATION and MPI_COMPILER ?
For example, if you want to use the PGI profiler pgprof. (More details ...)
How was my MPI-enabled executable generated ?
Unless you use only the system default compilers and MPI implementation, otherwise, you might at times wonder how a specific executable was generated. This is answered -- albeit indirectly -- by the ldd command:
```
katana:~ % ldd a.out
```
The output, which prints the paths of many library or command files, reveals whether it is "pgi" or "gcc" as well as openmpi or mvapich (mpich) and hence provide clues as to what compiler and what MPI implementation were used to generate the executable.

OIT | CCS | December 19, 2007