Running Jobs

On the pSeries, both serial and parallel jobs can be run either interactively (i.e., execute at the system prompt on the computer monitor) or submit to the batch queue for processing on the compute nodes. Jobs running interactively must not exceed approximately 10-minutes in CPU time per processor. All jobs expected to run longer than 10 minutes should be submitted to an appropriate batch queue. When the system determines that an interactive job has exceeded the 10-minute limit, the job will be terminated and a message alerts the owner of the job to that effect.

Batch queue technical summary

• Batch job scheduler is LSF. Common commands are: bsub, bjobs, and bkill.
• The amount of memory per processor is 1GB or 2GB depending on the specific machine (queue). The maximum agregate memory is 16 GB.
• The maximum number of processors a user can request is 8.
• The maximum wallclock time limit for multiprocessor queues are 5 hours. The maximum wallclock time limit for single processor queue is 80 hours.
• See the Technical Summary section for additional information.

How to run serial (single processor) jobs

• Interactive jobs
  Do not use mpirun. Simply type the executable name with any required file redirection, and specify the number of processors with the -procs flag.

  twister% a.out

• Batch jobs

  twister% bsub -q p4-long "a.out < myinputfile > myoutputfile"

  In the above example, LSF's bsub is used to submit the serial job to the pSeries serial p4-long batch queue. The colored items above denote optional I/O redirects. For information on queues, see the Technical Summary page.

How to run MPI jobs

• Interactive jobs
  The standard way to run multiprocessor MPI jobs on the pSeries is as follows:
  

  twister% a.out -procs 4

• Batch jobs

  twister% bsub -q p4-mp4 "a.out -procs 4"

  In the above example, LSF's bsub is used to submit a 4-processor job to the pSeries 4-processor p4-mp4 batch queue. For information on queues, see the Technical Summary page.

How to run OpenMP jobs

• Interactive jobs

  twister% setenv OMP_NUM_THREADS 4
  twister% a.out

• Batch jobs

  twister% setenv OMP_NUM_THREADS 4
  twister% bsub -q p4-mp4 a.out

Batch Job Management Commands

• To submit a batch job

  twister:~ % bsub . . .

• To query the status of batch jobs
  

  twister:~ % bjobs . . .

• To delete a batch job

  twister:~ % bkill . . .

• To display information about queues

  twister:~ % bqueues . . .

• To suspend a batch job

  twister:~ % bstop . . .

• To resume a batch job

  twister:~ % bresume . . .

• To charge a batch job to a project
  A batch job is normally charged to the user's default project. If the user works on a single project or if the charge should be levied against the default project, no user action is required. On the other hand, users working on multiple projects may, at times, need to charge a batch job to a non-default project. Note that the charging procedure varies among all SCV machines (See FAQ, Project Accounting). Please consult the respective machine's runningjobs webpage for the correct procedure. Described below are two charging procedures for the IBM pSeries.
  1. Primary method

     twister:~ % bsub -P project-name . . .

     Supplying the project-name through the bsub command is the most direct method. It supercedes any newgrp command you may have set previously.
  2. Alternative method

     twister:~ % newgrp project-name
     twister:~ % bsub . . .

  project-name will be charged for this and subsequent batch jobs (submitted from this window, or shell) until the next newgrp command is executed. Charging scheme for batch jobs submitted from another (non-descendant) window will not be affected.
• To find out the projects of which you are a member

  katana:~ % groups
  my_default_project  my_second_project  my_third_project . . .
  

  The first on the list is always the default project which can be changed.