Get an account¶
Do I have access?
Any employee with a NetID should be able to connect out-of-the-box. If you are a Master student, you have to request an account here. If you are a Bachelor student, you have to be a part of the course. For guest researchers, the procedure is described here.
The accounting of computing time on DelftBlue follows the fair share principle, as explained below. These faculties and institutions hold a share for research on the machine:
- CiTG/CEG 12%
- EWI/EEMCS 12%
- LR/AE 12%
- ME 12%
- TBM/TPM 12%
- TNW/AS 12%
- BK/ABE 6%
- IO/IDE 2%
- QuTech 2%
Resources dedicated to education, innovation, and individual projects:
- Education 9%
- Innovation 5%
- Projects 4%
On DelftBlue, slurm accounts are further subdivided into institutes per faculty.
In order to see to which accounts you have access, you can use the command
If the users of a share are using more computing time than is allocated to them, the priority of their jobs goes down. Jobs may still run if other parties are under-using their computing time (so that some cluster resources are still idle and available for these jobs).
Putting your CPU hours request into perspective
DelftBlue Phase 1 has 218 nodes with 48 CPUs each, and Phase 2 has 90 nodes with 64 CPUs each. That's a total of \((218*48 + 90*64)*24*365 =144\) M CPU hours a year for the entire university. If your faculty has 12% fair share, this comes down to ~17 M CPU hours a year for the entire faculty.
To make sure that everyone gets a fair chance of running their job, there are some limits associated with different account types. To see which limits are associated with your accounts, use the following command:
sacctmgr list user $USER withassoc format='user%-20,account%-45,maxjobs,maxsubmit,maxwall,maxtresperjob%-40'
How to request a correct account for your job¶
In order to request a correct account for your Slurm job, the following line needs to be added to the submission script:
The following accounts are available:
Innovation share¶
This is the default account for all TU Delft employees (i.e. it is assumed if the --account= line is absent from the Slurm submission script). This account gives you a relatively low priority, and only allows requesting limited amount of resources per job.
Note
The default "innovation" account has very restrictive limits implemented: no more than 4 jobs running simultaneously per user, no more than 64 CPUs, 2 GPUs and 250GB of memory (over 2 nodes maximum) can be requested per job, and a job can not run longer than 24 hours. No more than 10 jobs can be in the queue per user simultaneously. Furthermore, no more than 8 GPUs can be occupied in total by all users in the entire innovation account.
For example, if you see the following reasons for your jobs pending: (AssocGrpCpuLimit) or (AssocMaxJobsLimit) - then you are hitting these limits. If this is a problem, then please request your research share via TopDesk!
Research share¶
Researchers (PhDs, postdocs, or "above") can request full access via a research account, through the Self Service Portal. The research shares are organized hierarchically, per faculty and per department:
For example, for researchers from EEMCS faculty and DIAM department, the correct account code is:
Note
The "reseach" accounts are very generous, but not unlimited: no more than 2400 CPUs, 8 GPUs and 10TB of memory (over 50 nodes maximum) can be requested per job, and a job can not run longer than 120 hours (on CPU-nodes) or 48 hours (on GPU-nodes). Additionally, no more than 64 jobs of the same user can run simultaneously.
The full list of available account codes is as follows:
ABE (BK)¶
#SBATCH --account=research-abe-aet
#SBATCH --account=research-abe-ar
#SBATCH --account=research-abe-mbe
#SBATCH --account=research-abe-ur
AE (LR)¶
#SBATCH --account=research-ae-asm
#SBATCH --account=research-ae-co
#SBATCH --account=research-ae-fpt
#SBATCH --account=research-ae-spe
AS (TNW)¶
#SBATCH --account=research-as-bn
#SBATCH --account=research-as-bt
#SBATCH --account=research-as-cheme
#SBATCH --account=research-as-imphys
#SBATCH --account=research-as-qn
#SBATCH --account=research-as-rid
#SBATCH --account=research-as-rst
CEG (CiTG)¶
#SBATCH --account=research-ceg-3md
#SBATCH --account=research-ceg-es
#SBATCH --account=research-ceg-grs
#SBATCH --account=research-ceg-gse
#SBATCH --account=research-ceg-he
#SBATCH --account=research-ceg-tp
#SBATCH --account=research-ceg-wm
EEMCS (EWI)¶
#SBATCH --account=research-eemcs-diam
#SBATCH --account=research-eemcs-ekl
#SBATCH --account=research-eemcs-ese
#SBATCH --account=research-eemcs-insy
#SBATCH --account=research-eemcs-me
#SBATCH --account=research-eemcs-qce
#SBATCH --account=research-eemcs-st
IDE (IO)¶
#SBATCH --account=research-ide-dos
#SBATCH --account=research-ide-hcd
#SBATCH --account=research-ide-sde
ME¶
#SBATCH --account=research-ME-bme
#SBATCH --account=research-ME-cor
#SBATCH --account=research-ME-dcsc
#SBATCH --account=research-ME-mse
#SBATCH --account=research-ME-mtt
#SBATCH --account=research-ME-pe
#SBATCH --account=research-ME-pme
QuTech¶
#SBATCH --account=research-qutech-qcd
#SBATCH --account=research-qutech-qid
#SBATCH --account=research-qutech-qrd
TPM (TBM)¶
#SBATCH --account=research-tpm-ess
#SBATCH --account=research-tpm-mas
#SBATCH --account=research-tpm-vti
Education share¶
This should be your default account, if you are a Master student and have been granted access to DelftBlue for your master thesis research. The education shares are organized hierarchically, per faculty and per specialization:
For example, for MSc students from EEMCS faculty and Applied Mathematics specialization, the correct account code is:
Note
The "education" accounts have relatively restrictive limits implemented: no more than 8 jobs running simultaneously per user, no more than 64 CPUs, 2 GPUs and 250GB of memory (over 2 nodes maximum) can be requested per job, and a job can not run longer than 24 hours. No more than 100 jobs can be submitted to the queue simultaneously. Furthermore, no more than 2000 CPUs, 12 GPUs and 10000GB of memory can be occupied in total by all users in the entire set of education accounts.
For example, if you see the following reasons for your jobs pending: (AssocGrpCpuLimit) or (AssocMaxJobsLimit) - then you are hitting these limits. If you need more resources, and it is scientifically justified, you might request being added to the faculty research share of your supervisor. Your supervisor should contact us, if this is the case.
The full list of available account codes is as follows:
MSc students¶
ABE (BK)¶
AE (LR)¶
AS (TNW)¶
#SBATCH --account=education-as-msc-ap
#SBATCH --account=education-as-msc-ce
#SBATCH --account=education-as-msc-lst
#SBATCH --account=education-as-msc-nb
#SBATCH --account=education-as-msc-qist
#SBATCH --account=education-as-msc-sec
#SBATCH --account=education-as-msc-set
CEG (CiTG)¶
#SBATCH --account=education-ceg-msc-aes
#SBATCH --account=education-ceg-msc-agp
#SBATCH --account=education-ceg-msc-ce
#SBATCH --account=education-ceg-msc-cme
#SBATCH --account=education-ceg-msc-envm
#SBATCH --account=education-ceg-msc-til
EEMCS (EWI)¶
#SBATCH --account=education-eemcs-msc-am
#SBATCH --account=education-eemcs-msc-ce
#SBATCH --account=education-eemcs-msc-cese
#SBATCH --account=education-eemcs-msc-cs
#SBATCH --account=education-eemcs-msc-dsaig
#SBATCH --account=education-eemcs-msc-ee
#SBATCH --account=education-eemcs-msc-es
#SBATCH --account=education-eemcs-msc-set
IDE (IO)¶
#SBATCH --account=education-ide-msc-dfi
#SBATCH --account=education-ide-msc-ipd
#SBATCH --account=education-ide-msc-spd
ME¶
#SBATCH --account=education-me-msc-be
#SBATCH --account=education-me-msc-me
#SBATCH --account=education-me-msc-mse
#SBATCH --account=education-me-msc-mt
#SBATCH --account=education-me-msc-ode
#SBATCH --account=education-me-msc-ro
#SBATCH --account=education-me-msc-sc
#SBATCH --account=education-me-msc-tm
TPM (TBM)¶
#SBATCH --account=education-tpm-msc-cosem
#SBATCH --account=education-tpm-msc-epa
#SBATCH --account=education-tpm-msc-ie
#SBATCH --account=education-tpm-msc-mot
BSc students¶
ABE (BK)¶
AE (LR)¶
AS (TNW)¶
#SBATCH --account=education-as-bsc-lst
#SBATCH --account=education-as-bsc-mst
#SBATCH --account=education-as-bsc-nb
#SBATCH --account=education-as-bsc-tn
CEG (CiTG)¶
EEMCS (EWI)¶
#SBATCH --account=education-eemcs-bsc-ee
#SBATCH --account=education-eemcs-bsc-ti
#SBATCH --account=education-eemcs-bsc-tw
IDE (IO)¶
ME¶
#SBATCH --account=education-me-bsc-kt
#SBATCH --account=education-me-bsc-mt
#SBATCH --account=education-me-bsc-wb
TPM (TBM)¶
Teaching courses on DelftBlue¶
If you need access to DelftBlue as a part of the course, then you can get the account code from your supervisor, or from the person responsible for the course you are participating in.
Course organizers
Please contact us so that we can create a correct course sub-group for you!
Which accounts do I have access to?¶
Q: What is the command on DelftBlue to see which accounts I have access to?
A: sacctmgr is a useful command to retrieve such information.
To get information per user:
You will see something like this:
[NetID@login02 ~]$ sacctmgr list user $USER withassoc format='user%-20,account%-45'
User Account
-------------------- ---------------------------------------------
NetID research-eemcs-diam
NetID innovation
What resources limits are associated with my accounts?¶
Q: What are the maximum resources I can use with my account?
A: To get this information, use the following command:
sacctmgr list user $USER withassoc format='user%-20,account%-45,maxjobs,maxsubmit,maxwall,maxtresperjob%-40'
You should see output summarizing the resources limits per account that you have access to:
User Account MaxJobs MaxSubmit MaxWall MaxTRES
-------------------- --------------------------------------------- ------- --------- ----------- ----------------------------------------
NetID education-eemcs-courses-delftblueworkshop 8 100 1-00:00:00 cpu=64,gres/gpu=2,mem=250G,node=2
NetID research-eemcs-diam 10000 5-00:00:00 cpu=2400,gres/gpu=8,mem=10000G,node=50
NetID innovation 4 10 1-00:00:00 cpu=64,gres/gpu=2,mem=250G,node=2
How do I get usage statistic for my username?¶
Q: What is the command on DelftBlue to see how much you as a user have used over the past time?
A: sreport is a useful command to retrieve such information.
To get information per user:
You should see output summarizing your usage, per account:
--------------------------------------------------------------------------------
Cluster/User/Account Utilization 2022-06-01T00:00:00 - 2023-03-30T23:59:59 (26179200 secs)
Usage reported in CPU Hours
--------------------------------------------------------------------------------
Cluster Login Proper Name Account Used Energy
--------- --------- --------------- --------------- -------- --------
delftblue dpalagin Dennis Palagin research-eemcs+ 14728 0
delftblue dpalagin Dennis Palagin innovation 3812 0
You can also get information on the overall usage of your faculty:
sreport cluster AccountUtilizationByAccount -t Hours start=2023-01-01 Accounts=research-eemcs | head -n 7
--------------------------------------------------------------------------------
Cluster/Account/User Utilization 2023-01-01T00:00:00 - 2023-03-30T23:59:59 (7686000 secs)
Usage reported in CPU Hours
--------------------------------------------------------------------------------
Cluster Account Login Proper Name Used Energy
--------- --------------- --------- --------------- ---------- --------
delftblue research-eemcs 1402616 0
Which jobs did I run recently?¶
You can use the sacct command to show all job information starting from a specific date:
[NetID@login02 ~]$ sacct -S 2022-12-01 -u $USER
JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
1853039 vnc-1147 visual innovation 2 FAILED 1:0
1853039.bat+ batch innovation 2 FAILED 1:0
1853040 vnc-1149 visual innovation 2 FAILED 1:0
1853040.bat+ batch innovation 2 FAILED 1:0
1853041 vnc-1150 visual innovation 2 FAILED 1:0
1853041.bat+ batch innovation 2 FAILED 1:0
1853042 vnc-1151 visual innovation 2 TIMEOUT 0:0
1853042.bat+ batch innovation 2 CANCELLED 0:15
1853751 all-OH compute research-+ 48 COMPLETED 0:0
1853751.bat+ batch research-+ 25 COMPLETED 0:0
1853751.0 aims.2107+ research-+ 48 COMPLETED 0:0
1853752 fewer-OH compute research-+ 48 COMPLETED 0:0
1853752.bat+ batch research-+ 12 COMPLETED 0:0
1853752.0 aims.2107+ research-+ 48 COMPLETED 0:0
1853754 no-OH compute research-+ 48 COMPLETED 0:0
1853754.bat+ batch research-+ 28 COMPLETED 0:0
1853754.0 aims.2107+ research-+ 48 COMPLETED 0:0
1856064 water compute research-+ 4 COMPLETED 0:0
1856064.bat+ batch research-+ 4 COMPLETED 0:0
1856064.0 aims.2107+ research-+ 4 COMPLETED 0:0
How do I check information on a specific job?¶
You can use the seff command to show the job details:
[NetID@login02 ~]$ seff 1853751
Job ID: 1853751
Cluster: delftblue
User/Group: dpalagin/domain users
State: COMPLETED (exit code 0)
Nodes: 2
Cores per node: 24
CPU Utilized: 22-15:49:25
CPU Efficiency: 98.96% of 22-21:32:00 core-walltime
Job Wall-clock time: 11:26:55
Memory Utilized: 20.83 GB (estimated maximum)
Memory Efficiency: 21.70% of 96.00 GB (2.00 GB/core)