Submit jobs

sbatch <run_script>.sh

Examples of job scripts

  • Serial job
#!/bin/bash
#SBATCH --account=def-afyshe-ab
# time (DD-HH:MM:SS)
#SBATCH --time=00-00:01:00
echo 'Hello, world!'
sleep 5
  • Array job
#!/bin/bash
#SBATCH --job-name=test
#SBATCH --account=def-afyshe-ab
#SBATCH --time=0-01:00:00
#SBATCH --mem-per-cpu=500M
# job name (%x), job ID (%j)
#SBATCH --output=output/%x-%j.txt
# run a 10 job array, with a maximum of 5 running at a time
#SBATCH --array=1-10%5
# job array with indexes [1,2,3,5,7]
#SBATCH --array=1,2,3,5,7

./myapplication $SLURM_ARRAY_TASK_ID

Interactive job

salloc --time=1:0:0 --ntasks=2 --account=def-someuser
salloc --time=0-01:00:00 --cpus-per-task=2 --account=def-afyshe-ab --mem-per-cpu=512M
salloc: Granted job allocation 1234567
...        # do some work
exit       # terminate the allocation
salloc: Relinquishing job allocation 1234567

Monitoring jobs

  • Show jobs for a specific user
squeue -u <user> -r
  • Show only running jobs, or only pending jobs
squeue -u <user> -t running
squeue -u <user> -t pending
PENDING, RUNNING, SUSPENDED, COMPLETING, COMPLETED, OUT_OF_MEMORY, FAILED
  • Show (detailed) information for a specific job
scontrol show job <jobid>
scontrol show job -dd <jobid>
  • Show status information for a running job
sstat -j <jobid>
# List info resource used by a job: Average cpu time, Max memory, Max virtual memory, Job ID
sstat -j <jobid> --format=AveCPU,MaxRSS,MaxVMSize,JobID
  • Attach a running job
srun --jobid=<jobid> --pty bash -i
  • Email notification
#SBATCH --mail-user=<email_address>
#SBATCH --mail-type=ALL
#SBATCH --mail-type=TIME_LIMIT
#SBATCH --mail-type=TIME_LIMIT_80

Completed jobs

  • Show a short summary of a completed job
seff <jobid>
  • Show a detailed summary of a completed job or all jobs of a user
sacct -j <jobid>
sacct -j <jobid> --format=JobID,JobName,AveCPU,MaxRSS,MaxVMSize,Elapsed
sacct –u <user> --format=JobID,JobName,AveCPU,MaxRSS,MaxVMSize,Elapsed

Controlling jobs

# Cancel a specific job
scancel <jobid>
# Cancel all jobs for a specific user
scancel -u $USER
# Cancel all pending jobs for a specific user
scancel -t PENDING -u $USER
# Cancel all running jobs for a specific user
scancel -t RUNNING -u $USER
# Cancel one or more jobs by name
scancel --name <jobName>
# Hold a job, prevent it form starting
scontrol hold <jobid>
# Release a job hold, allowing the job to try to start
scontrol release <jobid>
# Release a previously held job to begin execution
scontrol resume <jobid>
# Requeue a running, suspended or finished job into pending state
scontrol requeue <jobid>
# List running jobs by user
squeue -u <user> -ho %A -t RUNNING
# Set a new Timelimit a running (need admin privilege)/pending job
scontrol update jobid=<jobid> TimeLimit=<TimeLimit>
# Set other parameters for a job
scontrol update jobid=<jobid> Account=<account> CPUsPerTask=<count> MinMemoryCPU=<MB> Gres=<list>

SLURM Environment Variables

Environment Variable Description
SLURM_JOB_NAME User specified job name
SLURM_JOB_ID Slurm job id
SLURM_NNODES Number of nodes allocated to the job
SLURM_NTASKS Number of tasks allocated to the job
SLURM_ARRAY_TASK_ID Array index for the job
SLURM_ARRAY_TASK_MAX Total number of array indexes for the job
SLURM_MEM_PER_CPU Memory allocated per CPU
SLURM_JOB_NODELIST List of nodes on which resources are allocated to job
SLURM_JOB_CPUS_PER_NODE Number of CPUs allocated per node
SLURM_JOB_PARTITION List of Partition(s) that the job is in
SLURM_JOB_ACCOUNT Account under which this job is run

Account information

# List user and their default account (accounting group)
sacctmgr show user <user> withassoc
# Show usage info for user
sshare -l -U <user>
# Show usage info for all users under a specific account
sshare -l -A <account>_cpu --all

Cluster information

# Show idle node on cluster
sinfo --states=idle
# Show down, drained and draining nodes and their reason
sinfo -R
# Show detailed node info
sinfo --Node --long
# Show reservations on the cluster
scontrol show reservation 
# Show configuration descriptions
man slurm.conf
# Check configuration values
scontrol show config | grep Max
# Show job info on cluster
partition-stats

Software modules

# Show currently loaded modules
module list
# Search for a module (if listed)
module avail <name>
# Will give a little bit more info
module spider <name>
# Load a module
module load <moduleName>
# Unload a module
module unload <moduleName>
# Show commands in the module
module show <moduleName>

Disk usage

quota
quota --per_user
diskusage_report --per_user --all_users

References