Back to main "Getting Started" document

Submitting Jobs with SLURM

Once you have an executable program and are ready to run it on the compute nodes, you must create a job script that performs the following functions:

• Use job batch options to request the resources that will be needed (i.e. number of processors, run time, etc.), and
• Use commands to prepare for execution of the executable (i.e. cd to working directory, source shell environment files, copy input data to a scratch location, copy needed output off of scratch location, clean up scratch files, etc).

After the job script has been constructed you must submit it to the job scheduler for execution. The remainder of this section will describe the anatomy of a job script and how to submit and monitor jobs.

SLURM Batch Script Options

All jobs must be submitted via a SLURM batch script or invoking sbatch at the command line . See the table below for SLURM submission options.

SLURM submission script options

Option	Description
#SBATCH --job-name=YourJobName	Recommended: Assigns a job name. The default is the name of SLURM job script.
#SBATCH --partition=PartitionName	Recommended: Specify the name of the Partition (queue) to use. Use this to specify the default partition or a special partition i.e. non-condo partition to which you have access.
#SBATCH --ntasks=2	Required: The maximum number of tasks per job. Usually used for MPI jobs.
#SBATCH --ntasks-per-node=4	Optional: A more granular way of specifying the number of tasks. If used with --ntasks, this limits the maximum number of tasks that will be assigned to each node. If used with --nodes instead of --ntasks, specifies the exact number of tasks that  will be run on each node (so total number of tasks = tasks-per-node X nodes). Run the command man sbatch for more details.
#SBATCH --cpus-per-task=16	Recommended: The number threads per task. Usually used for OpenMP or multi-threaded jobs. If hyperthreading is ON, then SLURM allocates N/2 cores, allowing two threads to execute on each core. If hyperthreading is OFF, SLURM allocates N cores, distributing one thread to each. See also --threads-per-core, below.
#SBATCH --threads-per-core=1	Optional:  Use this to turn OFF hyperthreading.  Without this line, SLURM will allocate two threads per core.  Recommended for CPU-bound computations (as opposed to memory-bound or IO-bound).
#SBATCH --time=08:00:00	Required: The maximum run time needed for this job to run, in days-hh:mm:ss.
#SBATCH --mem-per-cpu=1024M	Recommended: The maximum amount of physical memory used by any single process of the job in ([M]ega|[G]iga|[T]era)Bytes. The value of mem-per-cpu multiplied by cpus on a node (mem-per-cpu X ntasks X cpus-per-task) should not exceed the amount of memory on a node.
#SBATCH --mem=2G	Optional: The maximum amount of memory per node used by all processes on that node, in ([M]ega|[G]iga|[T]era)Bytes. If your job is going to use the entire node, then you should use the --exclusive option instead of the --mem or --ntasks-per-node options. See this FAQ for more details.
#SBATCH --mail-user=netID@rice.edu	Recommended: Email address for job status messages. Replace netID with your netID as Rice.
#SBATCH --mail-type=ALL	Recommended: SLURM will notify the user via email when the job reaches the following states BEGIN, END, FAIL or REQUEUE.
#SBATCH --nodes=1 --exclusive	Optional: Using both of these options will give your job exclusive access to a node such that no other jobs will use the unallocated resources on the node. Please see our FAQ for more details on exclusive access.
#SBATCH --output=mypath	Optional: The full path for the standard output (stdout) and standard error (stderr) "slurm-%j.out" file, where the "%j" is replaced by the job ID. Current working directory is the default.
#SBATCH --error=mypath	Optional: The full path for the standard error (stderr) "slurm-%j.out" files. Use this only when you want to separate (stderr) from (stdout). Current working directory is the default.
#SBATCH --export=ALL	Optional: Exports all environment variables to the job. See our FAQ for details.
#SBATCH --account=AccountName #SBATCH --partition=PartitionName	You need to specify the name of the condo account to use a condo on the cluster. Use the command sacctmgr show assoc user=netID to show which accounts and partitions with which you have access. (Replacing netID with your actual Rice NetID.)
#SBATCH --constraint=<feature-list>	Optional: Constrains job to nodes matching a feature list. Currently available features include: processor architectures: cascadelake, icelake, sapphirerapids, or milan fabrics: opath, or ib (for Omni-Path or InfiniBand, respectively) Features can be combined: --constraint="icelake&opath"
#SBATCH --gres=gpu:1 or (for example) #SBATCH --gres=gpu:volta:1	Optional: Request a number of GPUs per node. If you do not specify the GPU type, SLURM will allocate the first available GPU(s). If you specify the GPU type, SLURM will allocate only that specific kind of GPU to your job: volta for Tesla V100 ampere for Tesla A40 lovelace for Tesla L40S

Serial Job Script

A job script may consist of SLURM directives, comments and executable statements. A SLURM directive provides a way of specifying job attributes in addition to the command line options. For example, we could create a myjob.slurm script this way:

myjob.slurm

This example script will submit a job to the default partition using 1 processor and 1GB of memory per processor, with a maximum run time of 30 minutes.

If you need to debug your program and want to run in interactive mode, the same request above could be constructed like this (via the srun command):

srun --pty --partition=interactive --ntasks=1 --mem=1G --time=00:30:00 $SHELL

For more details on interactive jobs, please see our FAQ on this topic.

SLURM Environment Variables in Job Scripts

When you submit a job, it will inherit several environment variables that are automatically set by SLURM. These environment variables can be useful in your job submission scripts as seen in the examples above. A summary of the most important variables are presented in the table below.

Important Environment Variables

Variable Name	Description
$SHARED_SCRATCH	Location of shared scratch space. See our FAQ for more details.
$LOCAL_SCRATCH	Location of local scratch space on each node.
$SLURM_JOB_NODELIST	Environment variable containing a list of all nodes assigned to the job.
$SLURM_SUBMIT_DIR	Path from where the job was submitted.

Job Launcher (srun)

For jobs that need two or more processors and are compiled with MPI libraries, you must use srun to launch your job. The job launcher's purpose is to spawn copies of your executable across the resources allocated to your job. We currently support srun for this task and do not support the mpirun or mpiexec launchers. By default srun only needs your executable, the rest of the information will be extracted from SLURM.

The following is an example of how to use srun inside your SLURM batch script. This example will run myMPIprogram as a parallel MPI code on all of the processors allocated to your job by SLURM:

myMPIjob.slurm

This example script will submit a job to the default partition using 24 processor cores and 1GB of memory per processor core, with a maximum run time of 30 minutes.

The following example will run myMPIprogram on only four processors even if your batch script requested more than four.

srun -n 4 /path/to/myMPIprogram

To ensure that your job will be able to access an MPI runtime, you must load an MPI module before submitting your job as follows:

module load GCC OpenMPI

Submitting and Monitoring Jobs

Once your job script is ready, use sbatch to submit it as follows:

sbatch /path/to/myjob.slurm

This will return a jobID number while the output and error stream of the job will be saved to one file inside the directory where the job was submitted, unless you specified otherwise.

The status of the job can be obtained using SLURM commands. See the table below for a list of commands:

SLURM Job Status Commands

Command	Description
squeue	Show a detailed list of all submitted jobs.
squeue -j <jobID>	Show a detailed description of the job given by <jobID>.
squeue --start -j <jobID>	Gives an estimate of the expected start time of the job given by <jobID>.
squeue -u <username>	Show a list of all jobs in queue owned by the user specified by <username> (i.e. NetID).
scontrol show job <jobID>	To get a verbose description of the job given by <jobID>. The output can be used as a template when you are attempting to modify a job.

There are many different states that a job can be after submission: BOOT_FAIL (BF), CANCELLED (CA), COMPLETED (CD), CONFIGURING (CF), COMPLETING (CG), FAILED (F), NODE_FAIL (NF), PENDING (PD), PREEMPTED (PR), RUNNING (R), SUSPENDED (S), TIMEOUT (TO), or SPECIAL_EXIT (SE). The squeue command with no arguments will list all jobs in their current state. The most common states are described here:

• Running (R): These are jobs that are running.
• Pending (PD): These jobs are eligible to run, but there is simply not enough resources to allocate to them at this time.

Deleting Jobs

A job can be deleted by using the scancel command as follows:

scancel jobID