Scientific Computing & Visualization | Computation | Katana Cluster

This Site

Programming for the Katana Cluster

On the Katana Cluster, the compilers (for C and FORTRAN) available are from GNU and PGI (Portland Group Inc.). All GNU compilers start with the letter g (e.g., g77 for GNU FORTRAN 77) while PGI compilers begin with "pg" (e.g., pgcc). For MPI programs, special scripts, such as mpiCC for MPI-enabled C++ codes, are available for the compilation of programs in these languages.

Running short jobs interactively on the login node Katana are permitted to facilitate and expedite users' program development and debugging. The interactive run procedures for various serial and parallel processing jobs are demonstrated.

Compilers

(Sixty-four-bit compilation is the system default. 32-bit compilation is optional.)

Language	Developer	Serial (Default¹)	File Suffixes	MPI²	OpenMP
Fortran 77	GNU	g77 (f77)	.f,.F,.for, .fpp .FOR,.FPP	mpif77	gfortran -fopenmp³
	PGI⁴	pgf77	.f, .F, .for .FOR .fpp, .FPP	mpif77	pgf77 -mp
	PGI	pghpf	.f, .F, .for .FOR .fpp, .FPP	N/A	pghpf -mp
Fortran 95	GNU	gfortran (f95)	.f,.f90	mpif90	gfortran -fopenmp
	PGI	pgf95 ⁵	.f,.f90	mpif90	pgf95 -mp
	PGI	pghpf	.f90,.f95,.F90,.F95	N/A	pghpf -mp
C	GNU	gcc (cc)	.c	mpicc	gcc -fopenmp
	PGI	pgcc	.c	mpicc	pgcc -mp
C++	GNU	g++ (c++)	.c	mpiCC⁶	g++ -mp
	PGI	pgCC	.c, .cc .cpp, .C	mpiCC	pgCC -mp

¹ For all languages shown, GNU compilers are the system default compilers. You may either use the GNU name or the generic name (shown in parentheses) for the respective languages.
² Please consult the MPI section below for details on the scripts such as mpif77 and mpicc.
³ Use gfortran to compile FORTRAN 77 program with OpenMP directives. It accepts all FORTRAN 77 OpenMP directive bindings and syntaxes, such as C$OMP.
⁴ Portland Group Inc.
⁵ You can also use pgf90, which is aliased to pgf95.
⁶ mpicxx and mpic++ are two aliases for mpiCC.

Compile a program parallelized with OpenMP directives

Examples for Portland Group Compilers

katana:~ % pgf77 -mp -o myexec myprogram.f

katana:~ % pgf95 -mp -o myexec myprogram.f90

katana:~ % pgcc -mp -o myexec myprogram.c

Examples for GNU Compilers

katana:~ % gfortran -fopenmp -o myexec myprogram.f

katana:~ % gfortran -fopenmp -o myexec myprogram.f90

katana:~ % gcc -fopenmp -o myexec myprogram.c

Additional OpenMP information ...
- If you use gfortran for OpenMP applications, care must be exercised with OpenMP directives' placement. (more details ...)
- OpenMP Timing comparisons of the IBM pSeries and Katana.

Running interactive OpenMP jobs

For program development and debugging purposes, short OpenMP jobs may run on Katana, the login node. These jobs are limited to 4 processors and 10 minutes of CPU time per processor.

Set processor count via the environment variable OMP_NUM_THREADS
```
katana:~ % setenv OMP_NUM_THREADS 2
```
Run job on katana
```
katana:~ % a.out
```

Compile a program parallelized with MPI

Compiling an MPI program is made easy with the use of the scripts mpif77, mpif90, mpicc and mpiCC (see table above). Demonstrated below are examples of using these compiler wrapper scripts to compile FORTRAN 77/90/95, C, and C++ programs.

katana:~ % mpif77 -o myexec myprogram.f

katana:~ % mpif90 -o myexec myprogram.f90

katana:~ % mpicc -o myexec myprogram.c

katana:~ % mpiCC -o myexec myprogram.C

The generic script mpif77, for example, by default uses the pgi FORTRAN 77 compiler together with the MPI library of the openmpi MPI implementation to compile and link the fortran program.

For MPI options, please consult, for example, the mpicc manpage. For code optimizations and other language-specific options, please refer to the PGI compilers' manpages, such as pgf95 (see table above).

Additional MPI information ...

MPI Timings for the Katana Cluster and other SCV computer systems.

MPI_IMPLEMENTATION and MPI_COMPILER environment variables

As an alternative to the system default, you may choose the mpich MPI implementation or select the gnu group of compilers for MPI code compilations. If you are interested in using these non-default options, please follow these instructions.

Running interactive MPI jobs

For program development and debugging purposes, short MPI jobs may run on Katana, the login node. These jobs are limited to 4 processors and 10 minutes of CPU time per processor.

katana:~ % mpirun -np 2 a.out

Caution: If you use non-default settings for environment variables MPI_IMPLEMENTATION or MPI_COMPILER, make sure that the executable was generated under the same conditions. (More details ...)

Compiling 32-bit executables

By default, all of the compilers on the Katana Cluster produce 64-bit executables. Additional compiler flags are required to build 32-bit executables:

Compiler Family	32-bit compilation flags
PGI	-tp k8-32
GNU	-m32

The mpi** wrapper scripts build 64-bit MPI programs by default. To build 32-bit MPI programs, use the flags appropriate to the underlying compiler, as specified by the MPI_COMPILER environment variable, with the wrapper scripts:

for the PGI compilers


katana:~ % katana:~ % setenv MPI_COMPILER pgi
katana:~ % mpicc -tp k8-32 myexample.c

for the GNU compilers


katana:~ % setenv MPI_COMPILER gnu
katana:~ % mpicc -m32 myexample.c

Scratch space usage

The scratch disks are available as temporary storage space. They are open to all SCF users and there is no preset quota for use. Files stored on any scratch disk are NOT BACKED UP and these files can only be kept in scratch for up to 10 days. Files not removed from scratch by the owner will be deleted by the system after 10 days. You can access a specific scratch disk via the pathname "/net/katana-aNN/scratch", where NN is a node number among 01, 02, ..., 13. It is important that you type the full path and not rely on the system to complete the word with tabbing.


katana:~ % cd /net/katana-a05/scratch      this works !

katana:~ % cd /net/katana-a05/scr<tab>     this fails !

The reason that the use of tab fails is because the node's scratch may not be mounted at the time. If it is not mounted, then as far as the system is concerned, "scratch" does not exist. Typing the key "katana-aNN/scratch" causes the system to automount katana-aNN's scratch.

The Katana login and compute nodes each has 50 GB of scratch disk space.
On a compute node, a reference to /scratch by default points to the local node's /scratch.
Similarly, if you are on the login node, type "cd /scratch" to access its own scratch.
If you'd like to use scratch space in a batch job, please use the scratch space of the compute nodes assigned to the job.
If you'd like to perform I/O on scratch while running a job on the login node, the katana login node's scratch may be used.