Running fMRIPrep on the SCC

fMRIPrep is far too heavy to run on a login node — always submit it to the queue. A single subject typically takes hours, so plan for a long job.

Unlike MRIQC, fMRIPrep has no group level. Every subject is preprocessed independently, which makes a job array the natural way to run a study: one task per participant, all running at the same time, so the study takes about as long as its slowest subject rather than the sum of them all.

The SCC uses the SGE scheduler, but many other clusters — and much of the field — use SLURM. Every scheduler-specific block below has a toggle, so you can read the same job as either. The fMRIPrep command itself never changes; only the directives and the submit commands around it do. Your choice is remembered as you move between blocks and pages.

For what the fMRIPrep options themselves mean, see Running fMRIPrep. For how many cores and how much memory to ask for, see the Appendix.

Participant Level

1. Copy the script and the participant list

[scc4]$ cp /project/scv/examples/imaging/fmriprep/fmriprep.qsub .
[scc4]$ cp /project/scv/examples/imaging/fmriprep/participant_list.txt .

[scc4]$ cp /project/scv/examples/imaging/fmriprep/fmriprep.sbatch .
[scc4]$ cp /project/scv/examples/imaging/fmriprep/participant_list.txt .

participant_list.txt holds one participant per line and must sit in the same directory as the script:

sub-01
sub-02
sub-03

Build it from your own dataset:

[scc4]$ ls -d /projectnb/your_project/bids/sub-* | xargs -n1 basename > participant_list.txt

2. Set the number of tasks

The #$ -t 1-3#SBATCH --array=1-3 line must match the number of participants in the list. Count them:

[scc4]$ wc -l participant_list.txt

If the range is too short, participants at the end are silently skipped; if it is too long, the extra tasks fail with an empty participant label.

3. Fill in the script

#!/bin/bash -l

# Set your SCC project
#$ -P your_project

# Specify hard time limit for the job. This applies to EACH task in the array.
#   The task will be aborted if it runs longer than this time.
#   The default time is 12 hours
#$ -l h_rt=12:00:00

# Send an email when the job finishes or if it is aborted (by default no email is sent).
#$ -m ea

# Give job a name
#$ -N fmriprep

# Combine output and error files into a single file
#$ -j y

# Write the job logs here. This directory must already exist -- the scheduler
# will not create it, and the job fails immediately if it is missing.
#$ -o /projectnb/your_project/logs/

#$ -pe omp 8
#$ -l mem_per_core=4G

# Run one task per participant. Change 3 to the number of lines in participant_list.txt
#$ -t 1-3

# Keep track of information related to the current job
echo "=========================================================="
echo "Start date : $(date)"
echo "Job name : $JOB_NAME"
echo "Job ID : $JOB_ID.$SGE_TASK_ID"
echo "=========================================================="

# load the fmriprep module
module load fmriprep/25.2.5

# specify where on the SCC your BIDS formatted data lives
BIDS_DIR='/projectnb/your_project/bids'

# specify where on the SCC you want the preprocessed output to be written
OUTPUT_DIR='/projectnb/your_project/bids/derivatives/fmriprep'

# specify where on the SCC you want temporary files to be written. Defaulting here to your job's temporary directory
WORK_DIR=$TMPDIR

# pick out this task's participant. participant_list.txt must sit in the same
# directory as this script, with one participant per line
readarray -t PARTICIPANT_LIST < "participant_list.txt"
PARTICIPANT=${PARTICIPANT_LIST[$(($SGE_TASK_ID-1))]}

# request computational resources. Note that fMRIPrep's --mem-mb is in
# MEGABYTES, so this is 32 GB -- match it to what you requested above
# (-pe omp 8 at mem_per_core=4G)
NPROCS=$NSLOTS
NTHREADS=$NSLOTS
MEM=32000

# run fMRIPrep on this task's participant. Add --fs-no-reconall to skip
# FreeSurfer surface reconstruction, which roughly halves the runtime if you
# do not need surface output
fmriprep $BIDS_DIR $OUTPUT_DIR participant --participant-label $PARTICIPANT --work-dir $WORK_DIR --notrack --nprocs $NPROCS --omp-nthreads $NTHREADS --mem-mb $MEM

#!/bin/bash -l

# Set the account the job is charged to (SGE calls this the project)
#SBATCH --account=your_project

# Specify hard time limit for the job. This applies to EACH task in the array.
#   The task will be aborted if it runs longer than this time.
#SBATCH --time=12:00:00

# Send an email when the job finishes or if it fails. Add --mail-user=<address>
# if your site does not default to the submitting user.
#SBATCH --mail-type=END,FAIL

# Give job a name
#SBATCH --job-name=fmriprep

# Write the job logs here. This directory must already exist -- the scheduler
# will not create it, and the job fails immediately if it is missing.
# %A is the array job ID and %a the task ID. Naming no --error file means
# errors are written into this same file.
#SBATCH --output=/projectnb/your_project/logs/fmriprep-%A_%a.out

#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=4G

# Run one task per participant. Change 3 to the number of lines in participant_list.txt
#SBATCH --array=1-3

# Keep track of information related to the current job
echo "=========================================================="
echo "Start date : $(date)"
echo "Job name : $SLURM_JOB_NAME"
echo "Job ID : $SLURM_ARRAY_JOB_ID.$SLURM_ARRAY_TASK_ID"
echo "=========================================================="

# load the fmriprep module
module load fmriprep/25.2.5

# specify where your BIDS formatted data lives
BIDS_DIR='/projectnb/your_project/bids'

# specify where you want the preprocessed output to be written
OUTPUT_DIR='/projectnb/your_project/bids/derivatives/fmriprep'

# specify where you want temporary files to be written. Not every SLURM site
# defines TMPDIR on compute nodes -- check your local documentation
WORK_DIR=${TMPDIR:-/tmp}

# pick out this task's participant. participant_list.txt must sit in the same
# directory as this script, with one participant per line
readarray -t PARTICIPANT_LIST < "participant_list.txt"
PARTICIPANT=${PARTICIPANT_LIST[$(($SLURM_ARRAY_TASK_ID-1))]}

# request computational resources. Note that fMRIPrep's --mem-mb is in
# MEGABYTES, so this is 32 GB -- match it to what you requested above
# (8 cpus at 4G each)
NPROCS=$SLURM_CPUS_PER_TASK
NTHREADS=$SLURM_CPUS_PER_TASK
MEM=32000

# run fMRIPrep on this task's participant. Add --fs-no-reconall to skip
# FreeSurfer surface reconstruction, which roughly halves the runtime if you
# do not need surface output
fmriprep $BIDS_DIR $OUTPUT_DIR participant --participant-label $PARTICIPANT --work-dir $WORK_DIR --notrack --nprocs $NPROCS --omp-nthreads $NTHREADS --mem-mb $MEM

4. Create the log directory and submit

The scheduler will not create the log directory for you, and the job fails straight away if it is missing:

[scc4]$ mkdir -p /projectnb/your_project/logs
[scc4]$ qsub fmriprep.qsub

[scc4]$ mkdir -p /projectnb/your_project/logs
[scc4]$ sbatch fmriprep.sbatch

Each task writes its own log there, named fmriprep.o<job ID>.<task ID>fmriprep-<job ID>_<task ID>.out, so a failed task is easy to isolate. Watch the array with:

[scc4]$ qstat -u <your BU login name>

[scc4]$ squeue -u <your login name>