Programming for the IBM Blue Gene

The IBM Blue Gene is designed strictly for parallel applications. It is best suited for codes that scale well into hundreds or even thousands of processors. General application software such as MATLAB and Mathematica are not available on the Blue Gene. If you need to use such software, consider using the IBM pSeries or the Linux Cluster.

Our system consists of 2 front-end login nodes, lee.bu.edu and levi.bu.edu, and 1024 back-end compute nodes, each containing two processors. The login nodes are for compiling codes to generate executables for the back-end and submitting jobs to the back-end.

The main objective of this page is to provide you with information on compiling codes for running on the compute nodes. Please go to the Code Porting page for a list of DO's and DON'T's. If your code requires external packages (e.g., linear algebra subroutines ), go to the Packages page to find out what is available. On the Blue Gene, all multiprocessor jobs must be submitted to batch to run on the back-end. This topic is covered in the Running Jobs section.

Compiling  

On the Blue Gene system, parallel programming is based solely on MPI. OpenMP and other thread-based paradigms are not supported.

The front-end login nodes and the back-end compute nodes have different processors. Consequently, when compiling a code on the login nodes for running on the back-end, a special compiler is needed to generate the executable. These cross compilers are distinguished from the regular ones by the prefix "blrts_" and are listed in Table 1 below. In addition, several MPI header files and libraries are required for compiling and linking.

The MPI and various Blue Gene header files are in the directory /bgl/BlueLight/ppcfloor/bglsys/include/ . You will therefore need to include -I/bgl/BlueLight/ppcfloor/bglsys/includei with your compiler flags.

Every Blue Gene program must be linked with at least four libraries which are in the directory /bgl/BlueLight/ppcfloor/bglsys/lib/. The main four required libraries are libmpich.rts.a, libdevices.rts.a, libmsglayer.rts.a, and librts.rts.a.

A convenient and recommended way to compile MPI programs that incorporates the above required header files and libraries is by way of a makefile. Here is a template makefile and an example makefile.

For a more comprehensive list of cross compilers, including the GNU compilers, see this.

For C++, besides the four MPI libraries listed above, an additional library, libcxxmpich.rts.a, must also be linked. Furthermore, include MPI header file mpi.h (just like for C) where necessary, before stdio.h and iostream.h , if used. In addition, the following flag must be included during compilation

blrts_xlC -o myexec -DHAVE_MPI_CXX -O3  . . .  

Please see example makefile for more details.

If you want to compile/link serial codes to run on the login nodes, the regular set of compilers — similar to those available on twister (IBM p690) — are listed in Table 2.

Some important Fortran/C compiler flags:

• maximum optimization: -O5 (similar to Power4)
• Optimization levels:
  • default = none
  • -O = standard optimization
  • -O2 = same as -O
  • -O3 = more aggressive than -O
  • -O4 = -O3 + -qhot + -qipa=level=1
  • -O5 = -O4 + -qipa=level=2
• Architecture-specific switches
  • -qarch=440d (default)
    Each processor has two floating point units (FPU). By default, both are used. To use only one FPU, use -qarch=440. For details on usage of both FPUs, see this IBM document.
  • -qtune=440 (default)
    This setting is for both -qarch=440 and -qarch=440d
• C Preprocessor
  If your Fortran file contains cpp lines (like #define, #include, ...), there are several ways to make the compiler to recognize them:
  • use the suffix .F (supported by many compilers)
  • compile with -qsuffix=cpp=* (where * is your file suffix)
    If you also want to see how the compiler handles the cpp statements, also use "-d" on the compile line; if your file is called bar.f, a file Fbar.f will be created. This file contains, in addition to the Fortran part, the expanded cpp statements.
• Have compiler generate information
  • -qlist
  • -qsource
  • -qreport
• Array-bounds check
  • -C — run time array bounds checking
  • -qcheck — long form of -C

Programming Tools  

• Debugger
• Totalview parallel debugger. (totalview, Brief Usage Doc)
• Profiler
• gprof