(The Uniform Resource Locator for this World Wide Web page is
"http://scv.bu.edu/documentation/tutorials/MPI/")

Introduction

• Objectives of this Tutorial
  1. Introduces you to the fundamentals of MPI by ways of F77, F90 and C examples
  2. Shows you how to compile, link and run MPI code
  3. Covers additional MPI routines that deal with virtual topologies
  4. Cites references
• What is MPI ?
  1. MPI stands for Message Passing Interface and its standard is set by the Message Passing Interface Forum
  2. It is a library of subroutines/functions, NOT a language
  3. MPI subroutines are callable from Fortran and C
  4. Programmer writes Fortran/C code with appropriate MPI library calls, compiles with Fortran/C compiler, then links with Message Passing library
  5. Fortran 90 is not officially supported under MPI-1. It will be supported in MPI-2 in the near future. (Many codes that are written in F90 do work with MPI-1)
• Why MPI ?
  1. For large problems that demand better turn-around time (and access to more memory)
  2. For Fortran "dusty deck", often it would be very time-consuming to rewrite code to take advantage of parallelism. Even in the case of SMP, as are the SGI PowerChallengeArray and Origin2000, automatic parallelizer might not be able to detect parallelism.
  3. For distributed memory machines, such as cluster of Unix work stations, cluster of NT/Linux PCs, IBM pSeries.
  4. Maximize portability; works on distributed and shared memory architectures.

Preliminaries of MPI Message Passing

• In a user code, wherever MPI library calls occur, the following header file must be included:

  #include "mpi.h" for C code or

  include "mpif.h" for Fortran code

  These files contain definitions of constants, prototypes, etc. which are neccessary to compile a program that contains MPI library calls

• MPI is initiated by a call to MPI_Init. This MPI routine must be called before any other MPI routines and it must only be called once in the program.
• MPI processing ends with a call to MPI_Finalize.
• Essentially the only difference between MPI subroutines (for Fortran programs) and MPI functions (for C programs) is the error reporting flag. In fortran, it is returned as the last member of the subroutine's argument list. In C, the integer error flag is returned through the function value. Consequently, MPI fortran routines always contain one additional variable in the argument list than the C counterpart.
• C's MPI function names start with "MPI_" and followed by a character string with the leading character in upper case letter while the rest in lower case letters. Fortran subroutines bear the same names but are case-insensitive.

Basic MPI Routines Through Examples

There are essentially two different paradigms in MPI programming, SPMD (Single Program Multiple Data) and MPMD (Multiple Programs Multiple Data). The example programs shown below employ the SPMD paradigm, i.e., an identical copy of the same program is used for each of the processes. Examples for the less frequently used MPMD paradigm (in which more than one program are used among the processes) are demonstrated in a separate page.

• Example 1 (F77 version). Numerical Integration
  • Example 1.1 Parallel Integration with MPI_Send, MPI_Recv
  • Example 1.2 Parallel Integration with MPI_Isend, MPI_Irecv
  • Example 1.3 Parallel Integration with MPI_Bcast, MPI_Reduce
  • Example 1.4 Parallel Integration with MPI_Pack, MPI_Unpack
  • Example 1.5 Parallel Integration with MPI_Gather, MPI_Scatter
• Example 1 (C version). Numerical Integration
  • Example 1.1 Parallel Integration with MPI_Send, MPI_Recv
  • Example 1.2 Parallel Integration with MPI_Isend, MPI_Recv
  • Example 1.3 Parallel Integration with MPI_Bcast, MPI_Reduce
• A set of examples, ranging from the simple to the advanced, may be downloaded from the following site:
  ftp://info.mcs.anl.gov/pub/mpi/using
  The downloaded file is called examples.tar.Z.
• You may also download the examples in this tutorial.

Compilation and Execution

On the IBM p655 and p690, we currently support IBM's implementation of standard MPI. To see the compilation and job execution procedure for IBM's MPI, Click here.

At Boston University, we use LSF for batch processing. Through LSF, the batch submission script, bsub, permits user to enter the number of processors via the switch "-n". Do not use this option. Rather, specify the number processors through mpirun (for SGI machines) or poe (for the IBM pSeries) as shown in the above example.

machine-name% bqueues to find out what queues are available.

machine-name% bqueues -l p4short to find out specifics of the queue "mpshort".

Alternatively, if you prefer x-window displays, use machine-name% xlsbatch to find out the status of your (and others) job(s).

For more detail on batch queues, consult the Scientific Computing Facilities Technical Summary Page

Debugging MPI code with SGI's Case Vision Debugger (CVD)

1. To use cvd for MPI, it is far easier to use the MPICH MPI. Please refer to the above section on how to compile and execute MPI code using MPICH.
2. For debugging, make sure you compile with "-g" instead of any optimization flag, such as -O3.
3. At prompt, type "cvd executable-name" (e.g., % cvd a.out)
4. You should now see a couple of windows pop up. The main window should display the source file (it is easier to do this in the directory where all the files live).
5. At the top, there is a dialogue box with "command:" on the left. You should see the executable name, along with path in this box. Complete the command line with the number of procs you want to use. (e.g. command: /usr/mypath/a.out -np 4) Please see above on how to run jobs under MPICH.
6. Set traps by clicking at the left of the program source statements
7. At the top left of the main window, there is a "Admin" heading. Click and select "multiprocess view". A new window pops.
8. In the multiprocess view window, click "config/preferences". Check "Attach to forked processes" and "Copy traps to forked processes" if they are not already checked. Click save and apply. This way, next time you won't have to do this step again.
9. you can now click "run" which you can find at the top right of main window. The execution will stop at the first trap.
10. You can go to Multiprocess View window to select a specific process. Clicking on "run" in main window will cause that process to continue execution to next stop (trap). Alternatively, you can click "continue all" or "step into all" or "step over all" to allow all processes to march on.
11. Finally, the "expression view" window permits inquiries of the values of variables in source code. Just type in the names.

CVD can also be used for performance analyses.

IBM pSeries Debugger

Communicators and Virtual Topology (this section under development)

In addition to the basic MPI routines demonstrated above, there are many other routines for various applications. Some of the more frequently used routines, grouped according to their functionalities, are discussed below:

• Collective Communications
• Communicators
• Virtual Topology Slide Presentation
• Virtual Topology
  • Example 2. Laplace Solver (in F90) using MPI virtual topology routines
  • Example 3. Matrix Transpose (in F90) using MPI virtual topology routines

Application of communicators and cartesian topology is demonstrated through a matrix transpose example

References

From book publishers :

1. Parallel Programming with MPI by P. S. Pacheco, Morgan Kaufmann, 1997
2. MPI: The Complete Reference by M. Snir, et. al., The MIT Press, 1996
3. Using MPI by W. Gropp, E. Lusk and A. Skjellum, The MIT Press, 1994

Online via the SGI Power Challege Array or Origin2000 machines:

1. There are man pages for MPI
2. MPI and PVM User's Guide
3. MPI: A Message-Passing Interface Standard

On the Internet:

1. Users' Guide to mpich, a Portable Implementation of MPI by P. Bridges et. al.
2. MPI: A Message-Passing Interface Standard. The postscript file of this document, /pub/mpi/mpi-report.ps, can be down-loaded via anonymous FTP from info.mcs.anl.gov
3. A User's Guide to MPI by Peter S. Pacheco. (This is a postscript file)
4. MPI chapter in Designing and Building Parallel Programs by Ian Foster
5. MPI Tutorial by EPCC, University of Edinburgh
6. MPI Tutorial by Cornell University
7. MPI Tutorial by the National Computational Science Alliance

Other Tutorials on the WEB:

1. AHPCC, University of New Mexico