// Deep Dive · HPC Administration

SLURM Workload Manager

📅 Updated: June 2026 ⏱ 9 min read 🏷 Job Scheduling · Cluster Management · HPC Operations

What is SLURM?

SLURM (Simple Linux Utility for Resource Management) is the world's most widely deployed HPC workload manager, used by more than 60% of TOP500 supercomputers including Frontier, Perlmutter, and LUMI. It manages the allocation of compute resources — nodes, CPUs, GPUs, memory — among competing jobs from multiple users.

SLURM operates as a daemon running on every node. The slurmctld controller daemon runs on the management node; slurmd runs on each compute node. Users submit jobs via command-line tools; SLURM queues them, schedules execution based on priority, and tracks resource usage for billing.

// Market position

SLURM is open-source (GPLv2), developed originally at Lawrence Livermore National Laboratory, and now maintained by SchedMD. It replaced PBS/Torque and LSF as the dominant HPC scheduler because of its scalability to millions of cores, active development, and strong community support.

Core Architecture

Component	Role	Location
slurmctld	Central controller — scheduling decisions, state management	Management / head node
slurmd	Node daemon — launches jobs, monitors resources, reports state	Every compute node
slurmdbd	Database daemon — accounting, job history, fairshare data	Database server
slurmrestd	REST API — programmatic cluster access	Management node

Essential SLURM Commands

Command	Purpose	Example
sbatch	Submit a batch job script	sbatch job.sh
srun	Run a parallel job interactively or within sbatch	srun -n 128 ./sim
squeue	View job queue status	squeue -u username
scancel	Cancel a queued or running job	scancel 12345
sinfo	View partition and node status	sinfo -l
sacct	Job accounting and history	sacct -j 12345 --format=JobID,Elapsed,CPUTime
scontrol	Admin control and job details	scontrol show job 12345
salloc	Allocate resources interactively	salloc -N 4 --time=1:00:00

Writing SBATCH Job Scripts

Jobs are submitted as shell scripts with SLURM directives in #SBATCH comments. Here is a complete GPU training job example:

#!/bin/bash
#SBATCH --job-name=llm-training
#SBATCH --partition=gpu-large
#SBATCH --nodes=8                    # 8 compute nodes
#SBATCH --ntasks-per-node=4          # 4 MPI ranks per node
#SBATCH --cpus-per-task=16           # 16 CPU threads per rank
#SBATCH --gpus-per-node=4            # 4 GPUs per node (32 total)
#SBATCH --mem=512G                   # 512 GB RAM per node
#SBATCH --time=24:00:00              # Max walltime: 24 hours
#SBATCH --account=project_12345      # Billing account
#SBATCH --output=logs/%j_stdout.log  # %j = job ID
#SBATCH --error=logs/%j_stderr.log
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=user@institute.edu

# Load modules
module load cuda/12.3 openmpi/4.1 python/3.11

# Set environment
export OMP_NUM_THREADS=16
export CUDA_VISIBLE_DEVICES=0,1,2,3

# Run the job
srun python train.py \
    --model llama-70b \
    --data /scratch/project_12345/dataset \
    --checkpoint-dir /project/project_12345/checkpoints
  

Partitions

Partitions (called queues in PBS) are groups of nodes with shared policies — time limits, access controls, and resource constraints. A cluster typically has multiple partitions for different use cases:

Partition example	Node type	Max time	Typical use
short	CPU nodes	4 hours	Testing, debugging, quick jobs
standard	CPU nodes	48 hours	Production simulation
long	CPU nodes	7 days	Long-running research jobs
gpu	GPU nodes	24 hours	AI training, GPU simulation
bigmem	High-memory nodes	48 hours	Genomics, in-memory databases
interactive	Mixed	2 hours	Interactive development

GPU Resource Allocation

SLURM 19.05+ introduced native GPU scheduling via GRES (Generic RESource). Requesting GPUs correctly is critical for AI workloads:

# Request specific GPU type
#SBATCH --gres=gpu:h100:4          # 4x H100 GPUs
#SBATCH --gres=gpu:a100:8          # 8x A100 GPUs

# Or use the newer --gpus directive
#SBATCH --gpus-per-node=4
#SBATCH --gpus-per-task=1          # 1 GPU per MPI rank

# Verify allocation inside the job
nvidia-smi
echo "CUDA devices: $CUDA_VISIBLE_DEVICES"
  

Job Arrays

Job arrays submit many similar jobs with a single sbatch command — ideal for parameter sweeps, ensemble runs, or processing large numbers of input files. SLURM assigns each task a unique SLURM_ARRAY_TASK_ID.

#!/bin/bash
#SBATCH --array=1-100              # 100 tasks, IDs 1-100
#SBATCH --array=0-999%20           # 1000 tasks, max 20 running at once

# Each task processes a different input file
INPUT_FILE="inputs/sample_${SLURM_ARRAY_TASK_ID}.dat"
OUTPUT_FILE="outputs/result_${SLURM_ARRAY_TASK_ID}.out"

python process.py --input $INPUT_FILE --output $OUTPUT_FILE
  

Quality of Service (QOS)

QOS policies layer additional constraints and priorities on top of partitions — used to implement fairshare scheduling, reservation policies, and special access for priority users or projects.

Priority boost — give certain projects higher scheduling priority
Resource limits — cap maximum nodes or GPUs a user can use simultaneously
Time overrides — allow some users to submit jobs with longer walltimes
Preemption — allow high-priority jobs to preempt lower-priority ones

Scheduling Algorithms

SLURM uses two main scheduling plugins:

Priority/Multifactor — ranks jobs by a weighted combination of age, fairshare, job size, QOS, and partition priority
Backfill — fills gaps in the schedule with smaller jobs that won't delay higher-priority jobs, dramatically improving cluster utilization

// Backfill in practice

A 1000-node job is waiting for resources. Backfill allows a 10-node job with a short walltime to run in the gap — provided it won't delay the large job. Well-tuned backfill scheduling routinely achieves 85–95% cluster utilization.

Accounting and Fairshare

SLURM's database daemon (slurmdbd) records every job's CPU-hours, GPU-hours, memory, and walltime. This enables:

Fairshare scheduling — users/projects who have used less than their allocation get higher priority
Billing — charge research projects for compute usage
Reporting — sreport generates utilization reports for cluster administrators

# View your fairshare score
sshare -u username

# Detailed accounting for completed jobs
sacct -u username --starttime=2026-01-01 \
    --format=JobID,JobName,Partition,AllocNodes,Elapsed,CPUTimeRAW,State

# Cluster-wide utilization report
sreport cluster utilization start=2026-01-01 end=2026-06-01
  

Best Practices for Users

Request accurate walltimes — overestimating delays your job in backfill scheduling; underestimating kills it before completion
Use scratch storage — read/write data from the fast parallel filesystem, not your home directory
Implement checkpointing — save state every 1–4 hours so long jobs can restart after failures
Test with small jobs first — verify correctness on 1–2 nodes before scaling to hundreds
Use job arrays for parameter studies — far more efficient than submitting hundreds of individual jobs
Monitor with squeue/sacct — check your jobs are actually running efficiently, not just queued

Key Takeaways

SLURM is the dominant HPC scheduler on over 60% of TOP500 systems
SBATCH scripts are the standard way to submit reproducible, documented HPC jobs
Partitions and QOS let administrators shape resource usage policies per project and user group
GPU-aware scheduling via GRES is essential for modern AI and simulation workloads
Backfill scheduling is the key to high cluster utilization — short, accurate walltime estimates help everyone