Understanding Job Priority and Fairshare¶

When DelftBlue is busy, Slurm has to decide which pending job starts next. This page explains how that order is determined, how to see where your job stands, and what you can do to get it started sooner.

Two mechanisms decide when a job starts:

• Priority: pending jobs are considered in order of their priority.
• Backfill: a lower-priority job may start earlier if it fits into a gap without delaying any higher-priority job. Whether a job fits depends on its size and its requested --time.

How priority is calculated¶

DelftBlue uses Slurm's multifactor priority plugin. Of the possible factors, only two are used:

The priority of a job is the sum of the two columns that sprio shows, minus the job's nice value (see Practical tips):

PRIORITY  =  FAIRSHARE                          +  AGE                          −  nice
          =  1,000,000,000 × fairshare factor   +  100,000,000 × age factor     −  nice

The other factors (JOBSIZE, PARTITION, QOS, ASSOC, TRES, SITE) have weight 0 and play no role.

How the weights work¶

Slurm first calculates each factor as a number between 0 and 1, and then multiplies it by the weight of that factor (PriorityWeightFairShare and PriorityWeightAge in the configuration below). sprio shows the result of that multiplication, so the weight is also the maximum a factor can add:

• The age factor is the time the job has been waiting divided by 14 days, up to 1. After 7 days of waiting, a job gets 0.5 × 100,000,000 = 50,000,000 for age; after 14 days or more, 100,000,000.
• The fairshare factor is close to 1 when your account and you have used little compared to your share, and close to 0 when you have used a lot (see Fairshare).

With these weights, a difference of 0.01 in the fairshare factor (0.01 × 1,000,000,000 = 10,000,000) is worth as much as about 1.4 days of waiting (0.1 × 100,000,000). Use sprio --weights to see the weights, and sprio --norm to see the factors themselves.

Fairshare mainly decides the order between accounts: jobs of an account that has used little compared to its share get a higher priority than jobs of an account that has used a lot. Waiting time still counts, though. Some research accounts use more than their share nearly all the time; their fairshare factor is then very low, and the order of their jobs is decided mainly by age. Because the age factor keeps growing while a job waits, in practice, jobs rarely wait longer than about 1.5 to 2 weeks.

Viewing the configuration¶

You can check the active configuration yourself:

scontrol show config | grep ^Priority

Example output (shortened):

[NetID@login02 ~]$ scontrol show config | grep ^Priority
PriorityDecayHalfLife   = 2-00:00:00
PriorityFlags           =
PriorityMaxAge          = 14-00:00:00
PriorityType            = priority/multifactor
PriorityUsageResetPeriod = NONE
PriorityWeightAge       = 100000000
PriorityWeightAssoc     = 0
PriorityWeightFairShare = 1000000000
PriorityWeightJobSize   = 0
PriorityWeightPartition = 0
PriorityWeightQOS       = 0
PriorityWeightTRES      = (null)

The Slurm multifactor priority documentation explains every parameter.

Priority of a pending job: sprio¶

sprio shows the priority of pending jobs and how it is built up from the factors. It shows nothing for running or finished jobs.

sprio --jobs=<jobid>

Example output:

[NetID@login02 ~]$ sprio --jobs=988746
          JOBID PARTITION   PRIORITY       SITE        AGE  FAIRSHARE
         988746 compute-p  904848863          0    5194114  899654750

Here the priority of 904,848,863 consists of 899,654,750 from fairshare (99.4%) and 5,194,114 from age (0.6%). Dividing by the weights gives the factors: a fairshare factor of 0.90 (899,654,750 ÷ 1,000,000,000) and an age factor of 0.052 (5,194,114 ÷ 100,000,000). The job has therefore been waiting for 0.052 × 14 days ≈ 17 hours. Only factors with a non-zero weight are shown as columns.

Other useful variants:

sprio --weights

Shows the weights of the factors:

[NetID@login02 ~]$ sprio --weights
          JOBID PARTITION   PRIORITY       SITE        AGE  FAIRSHARE
        Weights                               1  100000000 1000000000

sprio --user=$USER

Shows all your own pending jobs, with an extra USER column:

[NetID@login02 ~]$ sprio --user=$USER
          JOBID PARTITION     USER   PRIORITY       SITE        AGE  FAIRSHARE
         998419 compute-p  <netid>  850651520          0          0  850651520
         998420 compute-p  <netid>  850651520          0          0  850651520

sprio --norm --jobs=<jobid>

Shows the normalised factors (between 0 and 1) instead of the weighted values.

sprio --long --jobs=<jobid>

Long format: also shows the user, account, QOS and nice value.

sprio --sort=-y

Sorts your pending jobs by priority, highest first.

Where is my job in the queue?¶

You can only see your own jobs, not those of other users, so you cannot see your place in the queue. The best indication is the estimated start time from squeue --me --start or sbatch --test-only, see When will my job start?. Treat it as a rough estimate:

• Backfill (see the start of this page) can start a small, short job long before jobs with a higher priority, if it fits into a gap without delaying them. The smaller your job and the shorter its --time, the more likely this is.
• Priorities are recalculated every 5 minutes, so the order changes as jobs age and fairshare values change.

To get an idea of how busy a partition is, use sinfo. With --summarize it shows the number of nodes that are allocated (in use), idle, other (e.g. down or in maintenance) and the total:

[NetID@login02 ~]$ sinfo --partition=compute-p1 --summarize
PARTITION   AVAIL  TIMELIMIT   NODES(A/I/O/T) NODELIST
compute-p1*    up 5-00:00:00      218/0/0/218 cmp[001-218]

Here all 218 nodes are in use, so a new job has to wait for others to finish, unless it fits into a gap left on a node that is not fully used.

Fairshare¶

The accounting of computing time on DelftBlue follows the fair share principle: each faculty account, and the education and innovation accounts, has a share of DelftBlue, see Accounting and shares. Shares are relative: what counts is the share of your account compared to the shares of the other accounts, not a fixed number of CPU hours. The more you and your account have used recently compared to that share, the lower your fairshare factor, and the lower the priority of your jobs.

How your usage is counted¶

Usage is not counted in CPU hours only. For every second a job runs, it is charged a billing value, which is the weighted sum of what it allocated (not what it actually used):

For example, a job with 8 cores and 24 GB of memory on compute-p1 is billed 8 + 0.25 × 24 = 14 per second. A job with 1 core, 1 GB and 1 GPU on gpu-a100 is billed 1 + 0.25 + 300 ≈ 301. You can see the billing value of a job in the ReqTRES field (for a running job: AllocTRES) of scontrol show job <jobid>, and the weights of a partition with:

scontrol show partition compute-p1 | grep TRESBillingWeights

Usage decays over time with a half-life of 2 days (PriorityDecayHalfLife): usage from two days ago counts half, from four days ago a quarter, and so on. A period of heavy use therefore lowers your priority for a few days, not for months.

Viewing your fairshare: sshare¶

sshare

shows one line for each account you can use. You only see your own lines: the usage of your account as a whole, and that of other users, is not visible.

[NetID@login02 ~]$ sshare
Account                    User  RawShares  NormShares    RawUsage  EffectvUsage  FairShare
-------------------- ---------- ---------- ----------- ----------- ------------- ----------
innovation              <netid>          1    0.000068           0      0.000000   0.998829

Use sshare --long to add the columns NormUsage and LevelFS, and the running and accumulated TRES minutes.

The columns:

RawShares

The shares assigned to you within the account.

NormShares

Your shares as a fraction of the shares of all users at the same level, i.e. the part you are entitled to.

RawUsage

The decayed usage in billing-seconds (see How your usage is counted).

NormUsage (with --long)

RawUsage as a fraction of the total usage of the whole machine.

EffectvUsage

Your usage as a fraction of the usage of all users at the same level. Compare it with NormShares: higher means you used more than your share.

LevelFS (with --long)

NormShares divided by EffectvUsage. Above 1 means you have used less than your share; below 1 means more. inf means no usage at all.

FairShare

The fairshare factor used in the priority calculation, between 0 and 1.

How the fairshare factor is determined¶

DelftBlue uses Slurm's default Fair Tree algorithm (PriorityFlags is empty). It works through the account tree from the top:

1. At each level, accounts are ranked by their LevelFS. On DelftBlue the top level holds, among others, research, education and innovation; below research are the faculty accounts such as research-as.
2. The department accounts (e.g. research-as-imphys) have parent shares: they do not have a separate share, so effectively all users of a faculty share the faculty's share, each with an equal part.
3. Users are ranked within that group by their own LevelFS.
4. The resulting order of all users on the machine is converted into the FairShare factor: 1.0 for the user with the highest rank, lower values further down.

So your priority depends first on how much your faculty (or the education or innovation account) has used relative to its share compared to the others, and second on how much you have used compared to your colleagues in that account.

The Slurm Fair Tree documentation gives the full details.

Why is my job waiting?¶

The REASON column¶

squeue --me shows why each pending job is not running in the NODELIST(REASON) column. With --start it also shows an estimated start time; see When will my job start?.

The most common reasons on DelftBlue:

The full list is in the Slurm job reason codes.

All details: scontrol show job¶

scontrol show job <jobid> shows all scheduling information of a pending or recently finished job, see also How do I get detailed scheduling information on a job?. The fields most relevant for waiting jobs are:

• Priority and Nice
• JobState and Reason
• Dependency
• QOS and Partition
• StartTime: for a pending job, the current estimate of the start time
• ReqTRES and AllocTRES: the requested and allocated resources, including the billing value

Limits and QOS¶

Limits on the number of jobs, cores, GPUs, memory and wall time are set per association (the combination of user and account) and per account. The limits per account type are described in Accounting and shares. To see the limits that apply to you, including the allowed QOS:

sacctmgr show assoc user=$USER format=Account%-35,QOS%20,MaxJobs,MaxSubmit,MaxWall,MaxTRES%40

Example output (shortened):

[NetID@login02 ~]$ sacctmgr show assoc user=$USER format=Account%-35,QOS%20,MaxJobs,MaxSubmit,MaxWall,MaxTRES%40
Account                                              QOS MaxJobs MaxSubmit     MaxWall                                  MaxTRES
----------------------------------- -------------------- ------- --------- ----------- ----------------------------------------
innovation                            normal,reservation       1        10  1-00:00:00        cpu=64,gres/gpu=2,mem=250G,node=2
research-as-imphys                    normal,reservation      64      1000  5-00:00:00   cpu=2400,gres/gpu=8,mem=10000G,node=50

A QOS (quality of service) can carry its own limits and priority. To list them:

sacctmgr show qos format=Name%15,Priority,MaxWall,MaxTRESPU%35,MaxJobsPU,MaxSubmitPU

[NetID@login02 ~]$ sacctmgr show qos format=Name%15,Priority,MaxWall,MaxTRESPU%35,MaxJobsPU,MaxSubmitPU
           Name   Priority     MaxWall                           MaxTRESPU MaxJobsPU MaxSubmitPU
--------------- ---------- ----------- ----------------------------------- --------- -----------
         normal          0                                                                 10000
    reservation          0                                                                 10000
 gpu-a100-small          0                     cpu=4,gres/gpu=2,mem=32000M         1

On DelftBlue you normally do not need to choose a QOS:

• normal is used by default.
• reservation can only be used together with a reservation.
• gpu-a100-small is applied automatically to jobs in the gpu-a100-small partition, and limits each user to one running job.

All QOS have priority 0, so the QOS does not influence the priority of your job.

Step by step: analysing a waiting job¶

1. Find the reason. Run squeue --me and look at the REASON column.
2. Limit reached? If the reason mentions Assoc or QOS, compare your running jobs with your limits (sacctmgr show assoc user=$USER ...). The job will start when your other jobs finish.
3. Dependency or hold? Check the job it depends on, or release the hold.
4. Priority or Resources? Check the priority with sprio --jobs=<jobid> and your fairshare with sshare --long. A low FairShare value means your faculty, or you, have used a lot recently; this recovers within days.
5. Estimate the start time with squeue --me --start or the StartTime in scontrol show job <jobid>. Estimates change as other jobs finish early or higher-priority jobs are submitted.
6. Adjust if worthwhile. A smaller request, a shorter --time or a different partition may let the job start sooner. Use sbatch --test-only <job script> to compare estimated start times without submitting, see Can I get an estimate when my job will run?.

Practical tips¶

• Do not request more than you need. Every allocated core, GB of memory and GPU counts towards your fairshare usage, whether you use it or not, and larger jobs are harder to fit in. Use seff to check what your jobs actually used.
• Request a realistic --time. The backfill scheduler can only start your job early if it knows the job ends before resources are needed for higher-priority jobs. A time limit of about 20–50% above the expected runtime gives the best chance; see Choosing a realistic --time.
• Spread your usage. Because usage decays with a half-life of 2 days, a steady stream of jobs keeps your priority higher than submitting a very large batch at once.
• Choose the right partition. The waiting time can differ a lot between, for example, compute-p1 and compute-p2. Check the partitions your job can run on and compare with sbatch --test-only.
• Use the right account. Research accounts have a larger share than the innovation account; see Accounting and shares.

• Lower the priority of less urgent jobs with --nice. The nice value is subtracted directly from the priority, so given the weights above it needs to be large to have an effect. For example, --nice=100000000 has the same effect as 14 days of waiting time:

  #SBATCH --nice=100000000
  

  For a job that is already pending: scontrol update jobid=<jobid> nice=100000000. Only administrators can set a negative nice value.