PowerFlow¶
Please note: you have to be added to the PowerFlow user group to be able to use this software!
Submit a job¶
The submission script looks as follows:
module load slurm
module load powerflow
EXA_QSYSTEM_DIR=slurm; export EXA_QSYSTEM_DIR
EXA_QSYSTEM_NAME=MySlurm; export EXA_QSYSTEM_NAME
EXA_PRINT_QSUB_CMD=1; export EXA_PRINT_QSUB_CMD
exaqsub -s --nprocs 96 --slurm --slurm_sim --ib --submit_options "--mem-per-cpu=3G --time=12:00:00 -D . -A research-ae-fpt" --queue compute GA_0012_RECT2_1REFREG_COARSE_ONLYGRID_windtunnel.cdi
The script starts the PowerFLOW executable on the login node, which does some light job preparation on the login node and then issues srun whenever it needs to actually calculate stuff.
Monitoring the job¶
PowerFlow has a command exawatcher, which allows you to monitor the progress of your job, for example:
[dpalagin@login04 ~]$ module load powerflow
[dpalagin@login04 ~]$ exawatcher
PowerFLOW Release 6-2021-R6-VLES-RT-beta2 (/beegfs/apps/generic/powerflow/6-2021-R6-VLES-RT-beta2)
Wed Dec 14 11:41:37 CET 2022 exaqsub: GA_0012_RECT2_1REFREG_COARSE_ONLYGRID_windtunnel (sim) job submitted
Wed Dec 14 11:41:45 CET 2022 Simulator: Initialization
Wed Dec 14 11:41:48 CET 2022 Decomposer: Reading CDI
Wed Dec 14 11:41:59 CET 2022 Decomposer: Decomposing
Wed Dec 14 11:42:01 CET 2022 Decomposer: Decomposing scale 3
Wed Dec 14 11:42:05 CET 2022 Decomposer: Decomposing scale 2
Wed Dec 14 11:42:06 CET 2022 Decomposer: Decomposing scale 0
Wed Dec 14 11:42:07 CET 2022 Decomposer: Computing ghost microblocks
Wed Dec 14 11:42:11 CET 2022 Decomposer: Transferring to CP process
Wed Dec 14 11:42:14 CET 2022 Simulator: Initializing voxels [39% complete [11684792 of 29943864]]
Wed Dec 14 11:42:14 CET 2022 Simulator: Initializing voxels [53% complete [15870904 of 29943864]]
Wed Dec 14 11:42:14 CET 2022 Simulator: Initializing voxels [64% complete [19164088 of 29943864]]
Sending extra instructions to the job¶
PowerFlow has a command exasignal, which allows you to send extra instructions to your job, for example, to start writing a checkpoint file. PLease note that you might need to export some environment varibles first:
[dpalagin@login04 ~]$ EXA_QSYSTEM_DIR=slurm; export EXA_QSYSTEM_DIR
[dpalagin@login04 ~]$ EXA_QSYSTEM_NAME=MySlurm; export EXA_QSYSTEM_NAME
[dpalagin@login04 ~]$ EXA_PRINT_QSUB_CMD=1; export EXA_PRINT_QSUB_CMD
[dpalagin@login04 ~]$ exasignal -full_ckpt
PowerFLOW Release 6-2021-R6-VLES-RT-beta2 (/beegfs/apps/generic/powerflow/6-2021-R6-VLES-RT-beta2)
Sending "full_ckpt" signal to "sim" job (MySlurm job 1870082)
General PowerFlow info¶
--------------------------------------------------------------------------------------------
PowerFLOW commands
--------------------------------------------------------------------------------------------
- exawatcher: it can be launched after submitting the simulation, it prints the log file and allows to monitor the phase of the simulation (discretization, decomposition, simulation) and the timestep.
- exasignal options: it sends a message to the simulation and asks for the command specified by "options" to be executed. In particular:
- exasignal -full_ckpt: it asks the simulation to save a checkpoint file as soon as possible.
To compare the saving time, after submitting the simulation I run exawatcher waiting for the timesteps to appear, then I exit exawatcher (ctrl + C) and I launch exasignal -full_ckpt. For DelftBlue, before running exasignal
it is necessary to set the environmental variables as in the submission script:
EXA_QSYSTEM_DIR=slurm; export EXA_QSYSTEM_DIR
EXA_QSYSTEM_NAME=MySlurm; export EXA_QSYSTEM_NAME
EXA_PRINT_QSUB_CMD=1; export EXA_PRINT_QSUB_CMD
After launching these commands and then exasignal -full_ckpt, I launch exawatcher again to monitor the progress of the saving.
-------------------------------------------------------------------------------------------
General information about PowerFLOW simulation
--------------------------------------------------------------------------------------------
PowerFLOW runs three consecutive operations: discretization, decomposition, simulation.
The starting input file is the one with the .cdi extension, the .lgi file is generated by the discretization and is read by decomposition and then simulation.
These two operations must be launched with the same number of processors.