Scientific Computing & Visualization | Documentation | Tutorials

This Site

Multiprocessing by Message Passing MPI

Introduction
Preliminaries
Basics
Compilation
Topologies
References

Introduction

Objectives of this Tutorial
1. Introduces you to the fundamentals of MPI by ways of F77, F90 and C examples
2. Shows you how to compile, link and run MPI code
3. Covers additional MPI routines that deal with virtual topologies
4. Cites references
What is MPI?
1. MPI stands for Message Passing Interface and its standard is set by the Message Passing Interface Forum
2. It is a library of subroutines/functions, NOT a language
3. MPI subroutines are callable from Fortran and C
4. Programmer writes Fortran/C code with appropriate MPI library calls, compiles with Fortran/C compiler, then links with Message Passing library
5. Fortran 90 is not officially supported under MPI-1. It will be supported in MPI-2 in the near future. (Many codes that are written in F90 do work with MPI-1)
Why MPI?
1. For large problems that demand better turn-around time (and access to more memory)
2. For Fortran "dusty deck", often it would be very time-consuming to rewrite code to take advantage of parallelism. Even in the case of SMP, as are the SGI PowerChallengeArray and Origin2000, automatic parallelizer might not be able to detect parallelism.
3. For distributed memory machines, such as cluster of Unix work stations, cluster of NT/Linux PCs, IBM pSeries.
4. Maximize portability; works on distributed and shared memory architectures.

Preliminaries of MPI Message Passing

In a user code, wherever MPI library calls occur, the following header file must be included:

#include "mpi.h" for C code or

include "mpif.h" for Fortran code

These files contain definitions of constants, prototypes, etc. which are neccessary to compile a program that contains MPI library calls
MPI is initiated by a call to MPI_Init. This MPI routine must be called before any other MPI routines and it must only be called once in the program.
MPI processing ends with a call to MPI_Finalize.
Essentially the only difference between MPI subroutines (for Fortran programs) and MPI functions (for C programs) is the error reporting flag. In fortran, it is returned as the last member of the subroutine's argument list. In C, the integer error flag is returned through the function value. Consequently, MPI fortran routines always contain one additional variable in the argument list than the C counterpart.
C's MPI function names start with "MPI_" and followed by a character string with the leading character in upper case letter while the rest in lower case letters. Fortran subroutines bear the same names but are case-insensitive.

Basic MPI Routines Through Examples

There are essentially two different paradigms in MPI programming, SPMD (Single Program Multiple Data) and MPMD (Multiple Programs Multiple Data). The example programs shown below employ the SPMD paradigm, i.e., an identical copy of the same program is used for each of the processes. Examples for the less frequently used MPMD paradigm (in which more than one program are used among the processes) are demonstrated in a separate page.

Example 1 (F77 version). Numerical Integration
- Example 1.1 Parallel Integration with MPI_Send, MPI_Recv
- Example 1.2 Parallel Integration with MPI_Isend, MPI_Irecv
- Example 1.3 Parallel Integration with MPI_Bcast, MPI_Reduce
- Example 1.4 Parallel Integration with MPI_Pack, MPI_Unpack
- Example 1.5 Parallel Integration with MPI_Gather, MPI_Scatter
Example 1 (C version). Numerical Integration
- Example 1.1 Parallel Integration with MPI_Send, MPI_Recv
- Example 1.2 Parallel Integration with MPI_Isend, MPI_Recv
- Example 1.3 Parallel Integration with MPI_Bcast, MPI_Reduce
A set of examples, ranging from the simple to the advanced, may be downloaded from the following site:
ftp://info.mcs.anl.gov/pub/mpi/using
The downloaded file is called examples.tar.Z.
You may also download the examples in this tutorial.

Compilation and Execution

On the IBM p655 and p690, we currently support IBM's implementation of standard MPI. To see the compilation and job execution procedure for IBM's MPI, Click here.

At Boston University, we use LSF for batch processing. Through LSF, the batch submission script, bsub, permits user to enter the number of processors via the switch "-n". Do not use this option. Rather, specify the number processors through mpirun (for SGI machines) or poe (for the IBM pSeries) as shown in the above example.

machine-name% bqueues to find out what queues are available.

machine-name% bqueues -l p4short to find out specifics of the queue "mpshort".

Alternatively, if you prefer x-window displays, use machine-name% xlsbatch to find out the status of your (and others) job(s).

For more detail on batch queues, consult the Scientific Computing Facilities Technical Summary.

Debugging MPI code with SGI's Case Vision Debugger (CVD)

To debug your code using SGI's Case Vision Debugger (cvd): (you will need x-window for cvd to work)

To use cvd for MPI, it is far easier to use the MPICH MPI. Please refer to the above section on how to compile and execute MPI code using MPICH.
For debugging, make sure you compile with "-g" instead of any optimization flag, such as -O3.
At prompt, type "cvd executable-name"(e.g., % cvd a.out)
You should now see a couple of windows pop up. The main window should display the source file (it is easier to do this in the directory where all the files live).
At the top, there is a dialogue box with "command:" on the left. You should see the executable name, along with path in this box. Complete the command line with the number of procs you want to use. (e.g. command: /usr/mypath/a.out -np 4) Please see above on how to run jobs under MPICH.
Set traps by clicking at the left of the program source statements
At the top left of the main window, there is a "Admin" heading. Click and select "multiprocess view". A new window pops.
In the multiprocess view window, click "config/preferences". Check "Attach to forked processes" and "Copy traps to forked processes" if they are not already checked. Click save and apply. This way, next time you won't have to do this step again.
you can now click "run" which you can find at the top right of main window. The execution will stop at the first trap.
You can go to Multiprocess View window to select a specific process. Clicking on "run" in main window will cause that process to continue execution to next stop (trap). Alternatively, you can click "continue all" or "step into all" or "step over all" to allow all processes to march on.
Finally, the "expression view" window permits inquiries of the values of variables in source code. Just type in the names.

bulb CVD can also be used for performance analyses.

IBM pSeries Debugger

There are several parallel debuggers on the IBM pSeries. Refer to the IBM pSeries section for more details.

Communicators and Virtual Topology (this section under development)

In addition to the basic MPI routines demonstrated above, there are many other routines for various applications. Some of the more frequently used routines, grouped according to their functionalities, are discussed below: