Comsol

Comsol Multi-Physics is a finite element simulation software. It supports parallel execution via multi-threading (shared memory) and/or MPI (multiple tasks). See the Comsol Knowledge Base for details.

You can setup your simulation (geometry, physics, meshing) on your local machien and copy the configuration to DelftBlue for the parallel simulation, or you can run it on a visual node to do these preprocessing steps.

Multi-threaded example

Here is an example job script to run a simulation on 4 cores of a single node. If you want to try it out, the corresponding model file can be downloaded here

compsol_smp.slurm

#!/bin/sh
#
#SBATCH --job-name="comsol_smp"
#SBATCH --output="comsol_smp.out"
#SBATCH --partition=compute
#SBATCH --time=0:15:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=1G

module load comsol/6.1

# base directory is where the job script is located (and the job starts)
ADRESS=${PWD}
SCRATCH=/scratch/$USER/
RECOVERY="${SCRATCH}/comsol_recovery_smp"
MODELTOCOMPUTE="${ADRESS}/comsol_model"
INPUTFILE="${MODELTOCOMPUTE}.mph"
OUTPUTFILE="${MODELTOCOMPUTE}_output_smp.mph"
BATCHLOG="${MODELTOCOMPUTE}_smp.log"

mkdir $RECOVERY

comsol batch -mpibootstrap slurm \
             -tmpdir ${RECOVERY} \
             -prefsdir ${RECOVERY} \
             -configuration ${RECOVERY} \
             -inputfile ${INPUTFILE} \
             -outputfile ${OUTPUTFILE} \
             -batchlog ${BATCHLOG}

Notes

• The -mpibootstrap option instructs COMSOL to use the settings given to slurm.
• Make sure you have a recovery folder in your simulation folder where Comsol can put all its temporary files.
• Specify the -tmpdir, -prefsdir and -configuration options, otherwise Comsol may store too much data in your home folder. see Comsol knowledge base.

Using distribuged memory

We can take the exact same job script and replace --ntasks=1 by --ntasks=4, and set --cpus-per-task=1 to get a pure MPI run. Combinations are also possible (i.e., running multiple threads per MPI process). For the small example here, the overhead of multi-processing is too large to pay off, so you may see a slow-down compared to the multi-threaded run. Another way of using more resources is running multiple simulations in parallel for varying parameters, see below.

Parallel "parameter sweep"

If a parametric sweep is peformed, the -ntasks options can be set to a higher value. In the advanced settings of the parametric sweep the option "distribute parametric sweep\" should be checked.

Make sure to empty the mesh solution before uploading the .mph file, sometimes (with parametric sweeps) this can otherwise give problems.

Simulation results

The results of the simulation can found in the .output.mph file and inspected on your own machine or a visual node.