Checking Job Efficiency with seff¶

Slurm reserves exactly the cores, memory and time you request, whether your job uses them or not. Resources that sit idle are unavailable to other users, still count towards your fair share usage, and a job that asks for more than it needs usually waits longer in the queue. This page shows how to check what a finished job actually used, and how to use that to size your next job.

Note

The example output on this page was produced on DelftBlue with Slurm 26.05. Usernames have been replaced with <netid>. Your numbers will of course differ.

Basic usage¶

seff ("Slurm efficiency") summarises how much of the requested CPU time and memory a finished job used. It is available on the login nodes:

seff <jobid>

For a task of a job array, add the task ID:

seff <jobid>_<taskid>

You can find the job ID in the output of sbatch, in the name of the default output file (slurm-<jobid>.out), or with sacct (see Which jobs did I run recently?).

Reading the output¶

Example output for a job with 4 tasks of 1 core each, requesting 2 GB per core and 15 minutes:

[NetID@login02 ~]$ seff 998419
Job ID: 998419
Cluster: delftblue
User/Group: <netid>/domain users
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 4
CPU Utilized: 00:15:57
CPU Efficiency: 98.86% of 00:16:08 core-walltime
Job Wall-clock time: 00:04:02
Memory Utilized: 4.71 GB
Memory Efficiency: 58.84% of 8.00 GB (2.00 GB/core)
The task which had the largest memory consumption differs by 100.01% from the average task max memory consumption

Line by line:

State: The final state of the job and the exit code of the batch script. Common states are COMPLETED, FAILED (non-zero exit code), TIMEOUT (the job hit its --time limit), OUT_OF_MEMORY and CANCELLED. Exit code 0 means success; any other value comes from your script or program.
Nodes and Cores per node: The number of nodes and the number of cores allocated per node. Cores per node is simply the total number of allocated cores divided by the number of nodes, so for jobs with an uneven distribution it is an average. For a job with a single core, seff prints Cores: 1 instead of these two lines.
CPU Utilized: The total CPU time used by all processes of the job, summed over all cores (format [days-]hh:mm:ss).
CPU Efficiency: CPU Utilized divided by the core-walltime: the number of allocated cores times the wall-clock time. In the example: 4 cores × 4 minutes 2 seconds = 16 minutes 8 seconds of core-walltime, of which 15 minutes 57 seconds were used, i.e. 98.86%. All cores were kept busy.
Job Wall-clock time: The elapsed (real) time between the start and the end of the job. Compare it with the --time you requested: here about 4 of the 15 requested minutes were used.
Memory Utilized: An estimate of the peak memory use of the job. seff adds up the peak memory of every task in a job step, and reports the job step with the highest total. In the example, each of the 4 tasks used about 1.2 GB. Because the tasks do not necessarily reach their peaks at the same moment, this is an upper estimate rather than an exact measurement.
Memory Efficiency: Memory Utilized as a percentage of the allocated memory. Between brackets is the allocated memory per core (if you used --mem-per-cpu) or per node (if you used --mem). In the example, 58.84% is somewhat low (see Low memory efficiency).
Last line (only for jobs with several tasks): Despite the wording, the percentage is the ratio between the task with the highest peak memory and the average task peak, times 100. In the example, 100.01% means that all tasks used the same amount of memory. A large value means the work, or the data, is not evenly divided over the tasks.

Job arrays¶

Each task of a job array has its own job ID, which seff shows next to the array notation. Example output for a single-core array task:

[NetID@login02 ~]$ seff 998420_1
Job ID: 998538
Array Job ID: 998420_1
Cluster: delftblue
User/Group: <netid>/domain users
State: COMPLETED (exit code 0)
Cores: 1
CPU Utilized: 00:01:00
CPU Efficiency: 95.24% of 00:01:03 core-walltime
Job Wall-clock time: 00:01:03
Memory Utilized: 303.82 MB
Memory Efficiency: 29.67% of 1.00 GB (1.00 GB/core)

Check a few tasks of an array: if they process different input, their usage can differ.

Interpreting the results¶

Low CPU efficiency¶

A CPU efficiency well below 100% means that allocated cores were idle for a large part of the time. For example, a short MPI test job on 2 nodes that only started a few small test programs shows:

[NetID@login02 ~]$ seff 128082
...
Nodes: 2
Cores per node: 4
CPU Utilized: 00:00:38
CPU Efficiency: 4.52% of 00:14:00 core-walltime
Job Wall-clock time: 00:01:45
...

Typical causes:

More cores requested than the program uses. Many programs (and Python scripts) run on one core unless you explicitly make them parallel. Check whether your software can run in parallel at all; see the note in Submit Batch Jobs.
Threads not matched to the allocation. For OpenMP and other multithreaded programs, set the number of threads to the number of allocated cores, as in the OpenMP job example:
```
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
```
A large serial part. Pre- and post-processing, setup or a single-threaded solver phase keep all but one core idle. The more cores you request, the more this costs (Amdahl's law). Measure how the runtime changes with the core count before requesting many cores; the Fluent page shows what such a scaling test looks like.
Waiting for I/O. Time spent reading or writing files is not CPU time. See Optimise Your I/O for ways to reduce this.
Very short jobs or interactive sessions. Start-up time weighs heavily in short jobs, as in the example above. An interactive or Open OnDemand session, where most of the time is spent thinking or typing, will also show a very low efficiency.

As a rule of thumb, aim for a CPU efficiency above about 80% for production runs.

CPU efficiency above 100% or oversubscription¶

Jobs on DelftBlue are confined to the cores they were allocated, so the CPU time cannot really exceed the core-walltime; values marginally above 100% are rounding effects. The real problem is oversubscription: starting more threads or processes than allocated cores (for example OMP_NUM_THREADS larger than --cpus-per-task, or a library that starts one thread per core of the whole node). The CPU efficiency then looks fine, close to 100%, but the threads compete for the same cores and the job runs slower than it should. Make sure the number of threads matches $SLURM_CPUS_PER_TASK.

Low memory efficiency¶

If the memory efficiency is low, reduce --mem-per-cpu (or --mem) for the next run. A good target is that the peak memory use is roughly 70–85% of what you request: this leaves a margin for variation between runs, without reserving memory no-one uses. Keep in mind that the measured peak is a sample (see Limitations), so do not cut it too fine for jobs whose memory use fluctuates strongly.

In the example above, each core used about 1.2 GB of the 2 GB requested (58.84%). Requesting --mem-per-cpu=1.5G would bring the efficiency to about 78%.

Requesting more memory per core than the partition provides also blocks cores for other users: a node with all its memory allocated cannot run other jobs, even if most of its cores are idle.

`OUT_OF_MEMORY`¶

A job with state OUT_OF_MEMORY was killed because it used more memory than it requested. seff then shows a memory efficiency close to (or above) 100%. Increase the memory request; see Slurm out-of-memory (OOM) error.

Choosing a realistic `--time`¶

Compare Job Wall-clock time with the time limit you requested (also shown by sacct as Timelimit, see below). Request roughly 20–50% more than the actual runtime of a comparable job: enough margin to avoid a TIMEOUT, but not so much that the scheduler cannot fit your job into a gap. For the example above, which ran for 4 minutes, --time=00:06:00 would be sufficient instead of 15 minutes. Shorter time limits help your job start sooner, as explained in Understanding Job Priority and Fairshare.

Limitations¶

Keep the following in mind when you use seff:

No GPU information. seff only reports CPU and memory usage. To check how well your job uses its GPUs, see GPU usage.
Memory is sampled. Slurm records the memory use of a job every 30 seconds. Short memory peaks between two samples are missed, and jobs that run for less than a minute or so may show little or no memory use at all.

Only for finished jobs. The accounting data seff uses are only complete once the job has ended. For a running job, seff shows zero usage and a warning; use sstat instead (see Running jobs: sstat):

[NetID@login02 ~]$ seff 998419
...
State: RUNNING
...
CPU Utilized: 00:00:00
CPU Efficiency: 0.00% of 00:08:32 core-walltime
Job Wall-clock time: 00:02:08
Memory Utilized: 0.00 MB
Memory Efficiency: 0.00% of 8.00 GB (2.00 GB/core)
WARNING: Efficiency statistics can only be obtained after the job has ended as seff tool is based on the accounting database data.

Estimates for multi-task and multi-node jobs. Memory Utilized is the sum of the individual task peaks, which overestimates the real peak if the tasks do not peak at the same moment. It also does not include the memory of the batch script itself when that runs alongside an srun step. Cores per node is an average over the nodes.

More detail with `sacct`¶

sacct shows the same accounting data per job step. Choose the columns with --format:

sacct --jobs=<jobid> --format=JobID,State,Elapsed,Timelimit,AllocCPUS,TotalCPU,ReqMem,MaxRSS

Example output for the same job as above:

[NetID@login02 ~]$ sacct --jobs=998419 --format=JobID,State,Elapsed,Timelimit,AllocCPUS,TotalCPU,ReqMem,MaxRSS
JobID             State    Elapsed  Timelimit  AllocCPUS   TotalCPU     ReqMem     MaxRSS
------------ ---------- ---------- ---------- ---------- ---------- ---------- ----------
998419        COMPLETED   00:04:02   00:15:00          4  15:56.844         8G
998419.batch  COMPLETED   00:04:02                     4  00:00.093                11548K
998419.0      COMPLETED   00:04:01                     4  15:56.751              1234088K

The first line is the job as a whole; the lines below it are the job steps: .batch is the batch script itself, and .0, .1, … are the individual srun commands in the script. Here, nearly all CPU time was used in step .0, the srun command.
Elapsed and Timelimit show the actual runtime versus the requested --time (here: 4 minutes used of 15 minutes requested).
TotalCPU is the CPU time used; compare it with AllocCPUS × Elapsed.
ReqMem is the total memory requested for the job.
MaxRSS is the peak memory of the largest single task in a step. It is only shown on the job-step lines, not on the line for the job as a whole. For a job with several tasks per step, the total memory use can be up to MaxRSS times the number of tasks: here 4 × 1234088K ≈ 4.71 GB, the value seff reports.

Other useful columns are AveRSS (average task peak memory), NNodes, NTasks, NodeList and ExitCode. Run sacct --helpformat for the full list, and see Trouble-shooting your Job for listing your recent jobs.

Running jobs: `sstat`¶

For a job that is still running, sstat shows the live usage per job step. It needs a job step, so for a job script without srun use the .batch step:

sstat --jobs=<jobid>.batch --format=JobID,AveCPU,MaxRSS,MaxRSSTask,NTasks

For a job that uses srun, use the step number (e.g. <jobid>.0), or show all steps with --allsteps. Example output for the job above, 2 minutes after it started:

[NetID@login02 ~]$ sstat --jobs=998419.0 --format=JobID,AveCPU,MaxRSS,MaxRSSTask,NTasks
JobID            AveCPU     MaxRSS MaxRSSTask   NTasks
------------ ---------- ---------- ---------- --------
998419.0       00:01:59   1234016K          3        4
[NetID@login02 ~]$ sstat --allsteps --jobs=998419 --format=JobID,AveCPU,MaxRSS,NTasks
JobID            AveCPU     MaxRSS   NTasks
------------ ---------- ---------- --------
998419.batch   00:00:00      7908K        1
998419.0       00:01:59   1234016K        4

AveCPU is the average CPU time per task so far: after 2 minutes each task had used almost 2 minutes of CPU time, so the cores are fully busy. MaxRSS is the peak memory of the largest task so far, and MaxRSSTask the number of that task.

To watch your job directly on the compute node (for example with top or nvidia-smi), see Accessing allocated nodes.

GPU usage¶

For jobs on the GPU partitions, Slurm also records the GPU usage of each job step: gres/gpuutil is the GPU utilisation in percent and gres/gpumem the GPU memory used. They are part of the TRESUsageInMax (peak) and TRESUsageInAve (average) columns of sacct, and of sstat for a running job:

sacct --jobs=<jobid> --parsable2 --format=JobID,Elapsed,TRESUsageInMax

Example output for a job that ran a GPU program with srun:

[NetID@login02 ~]$ sacct --jobs=1004433 --parsable2 --format=JobID,Elapsed,TRESUsageInMax
JobID|Elapsed|TRESUsageInMax
1004433|00:02:32|
1004433.batch|00:02:32|cpu=00:00:00,energy=0,fs/disk=1735980,gres/gpumem=0,gres/gpuutil=0,mem=21784K,pages=0,vmem=0
1004433.0|00:02:30|cpu=00:02:29,energy=0,fs/disk=297427,gres/gpumem=4404M,gres/gpuutil=100,mem=27924K,pages=5,vmem=0

The values are recorded per job step, for the step that ran the GPU program: here step .0, the srun command, which kept the GPU 100% busy and used 4.4 GB of GPU memory. The line for the job as a whole stays empty, and the .batch step shows 0 unless your GPU program runs directly in the job script without srun.
Like memory, GPU usage is sampled every 30 seconds.
On gpu-a100-small, the values are always 0: the GPUs there are split into instances (MIG), and those do not report their usage.

To watch the GPUs while your job runs, use nvidia-smi on the node, see Accessing allocated nodes, or add it to your job script as shown in the GPU job example.

Recommended workflow¶

Run a short test job with a representative (but smaller or shorter) input and a generous request for memory and time.
Check the result with seff <jobid> and, if needed, sacct.
Adjust the request: the number of cores to what the program really uses, the memory to give a peak use of about 70–85%, and --time to about 20–50% above the measured runtime (scaled to the full problem size).
Check the scaling if you want to use more cores: run the same test on, for example, 12, 24 and 48 cores, and keep the core count at which the speed-up is still worth it.
Scale up to your production runs, and run seff on the first few of them to make sure the request is still right.