Skip to content

Load software modules

Note: this page contains some example output from the module command that may not be up-to-date, it is meant for illustration purposes only.

Default module stack

The default DelftBlue software stack is called 2023r1:

[<netid>@login01 ~]$ module avail

-------------------------------------------------------------------- /apps/noarch/modulefiles ------------------------------------------------------------
   2022r2    2023r1    2023r1-gcc11    2023r1-intel

The 2023r1 software stack should be used by default. The others are built using specific compilers in case you want to use these compilers and link against libraries from the module system (e.g., MPI, OpenBLAS or PETSc). The default stack can be loaded as follows:

module load 2023r1

Within the software stack, you will see two types of modules:

  1. modules that can be loaded for all nodes (/apps/generic/modulefiles);
  2. modules specific to compute or gpu nodes (/apps/arch/2023r1/lmod/linux-rhel8-x86_64/Core)

Different partitions may have different modules!

Modules you see when you execute module avail depend on which partition of DelftBlue you are on. compute and login nodes have their own collection of software, gpu nodes have their own collection of software, and visual nodes have yet another collection of software. For example, do not just assume that exactly the same modules with exactly the same version names are available both on login and gpu nodes!

--------------------------------------- /apps/noarch/modulefiles ---------------------------------------
   2023r1 (L)

-------------------------------------- /apps/generic/modulefiles ---------------------------------------
   abaqus/2022         feko/2022                      miniconda3/4.12.0
   ansys/2021R2        intel/oneapi-all_2022.3        mstar/solver_3.7.53
   comsol/5.6          intel/oneapi-all               mstar/3.7.53        (D)
   cuda/11.6    (D)    intel/oneapi                   nvhpc/22.3
   cuda/11.7           intel/psxe_2020         (D)    paraview/5.9.1      (D)
   desktop/vnc         ls-dyna/13.0.1                 powerflow/6.2021.R6
   diana/10.5          matlab/R2021b                  slurm/current       (L,D)

---------------------------- /apps/arch/2023r1/lmod/linux-rhel8-x86_64/Core ----------------------------
   alsa-lib/                       mkfontdir/1.0.7
   apr-util/1.6.1                         mkfontscale/1.1.2
   apr/1.6.2                              mpc/1.1.0
   autoconf-archive/2019.01.06            mpfr/4.1.0
   autoconf/2.69                          nasm/2.15.05
   automake/1.16.3                        nccl/2.9.9-1
   bazel/3.7.2                            ncurses/6.2
   bdftopcf/1.0.5                         ninja/1.10.2
   berkeley-db/18.1.40                    numactl/2.0.14
   binutils/2.37                          openjdk/11.0.12_7
   bison/3.8.2                            openmpi/4.1.1
   boost/1.77.0                           openssh/8.7p1
   bzip2/1.0.8                            openssl/1.1.1l
   c-ares/1.15.0                          pango/1.42.0
   c-blosc/1.21.0                         paraview/5.9.1
   cairo/1.16.0                           pcre/8.44
   cgal/4.13                              pcre2/10.36
   cmake/3.21.4                           perl-alien-svn/
   cpio/2.13                              perl-data-dumper/2.173
   cryptsetup/2.3.5                       perl-module-build/0.4224
   cuda/10.2.89                           perl/5.34.0
   cudnn/                    pixman/0.40.0
   curl/7.79.0                            pkgconf/1.8.0
   diffutils/3.8                          pmix/3.2.1
   doxygen/1.9.2                          popt/1.16
   eigen/3.4.0                            protobuf/3.15.7
   expat/2.4.1                            psimd/2020-05-17
   ffmpeg/4.3.2                           pthreadpool/2021-04-13
   findutils/4.8.0                        py-absl-py/0.13.0
   flatbuffers/1.12.0                     py-astunparse/1.6.3
   flex/2.6.4                             py-certifi/2021.10.8
   font-util/1.3.2                        py-cppy/1.1.0
   fontconfig/2.13.94                     py-cycler/0.10.0
   fontsproto/2.1.3                       py-cython/0.29.24
   freetype/2.11.0                        py-future/0.18.2
   fribidi/1.0.5                          py-gast/0.3.3
   fxdiv/2020-04-17                       py-google-pasta/0.2.0
   gawk/5.1.0                             py-grpcio/1.32.0
   gcc/11.2.0                             py-joblib/1.0.1
   gdbm/1.19                              py-keras-preprocessing/1.1.2-intel-mkl
   gettext/0.21                           py-kiwisolver/1.3.2
   ginkgo/1.4.0                           py-matplotlib/3.4.3-intel-mkl
   git-lfs/2.11.0                         py-numpy/1.19.5-intel-mkl
   git/2.31.1                             py-opt-einsum/3.3.0-intel-mkl
   glib/2.70.0                            py-packaging/21.0
   gmake/4.3                              py-pillow/8.0.0
   gmp/6.2.1                              py-pip/21.1.2
   gnuplot/5.4.2                          py-pkgconfig/1.5.1
   go-bootstrap/1.4-bootstrap-20171003    py-protobuf/3.17.3
   go/1.17.2                              py-pybind11/2.6.2
   gobject-introspection/1.56.1           py-pyparsing/2.4.7
   gperf/3.1                              py-python-dateutil/2.8.2
   gsl/2.7                                py-pyyaml/5.3.1
   harfbuzz/2.6.8                         py-scikit-learn/1.0.1-intel-mkl
   help2man/1.47.16                       py-scipy/1.5.2-intel-mkl
   hwloc/2.6.0                            py-setuptools-scm/6.3.2
   icu4c/67.1                             py-setuptools/58.2.0
   inputproto/2.3.2                       py-six/1.15.0
   intel-mkl/2020.4.304                   py-termcolor/1.1.0
   jasper/2.0.32                          py-threadpoolctl/2.0.0
   json-c/0.15                            py-tomli/1.2.1
   julia/1.6.3-intel-mkl                  py-tqdm/4.62.3
   kbproto/1.0.7                          py-typing-extensions/
   kim-api/2.2.1                          py-wheel/0.36.2
   krb5/1.19.2                            py-wrapt/1.12.1
   libaio/0.3.110                         python/3.8.12
   libbsd/0.11.3                          qhull/2020.2
   libcerf/1.3                            r/4.1.1
   libedit/3.1-20210216                   rdma-core/54mlnx1
   libevent/2.1.8                         readline/8.1
   libfabric/1.13.2                       renderproto/0.11.1
   libffi/3.3                             rust/1.51.0
   libfontenc/1.1.3                       scala/2.13.1
   libgd/2.2.4                            scons/3.1.2
   libgit2/1.1.1                          scrnsaverproto/1.2.2
   libgpg-error/1.42                      serf/1.3.9
   libiconv/1.16                          shadow/4.8.1
   libidn2/2.3.0                          slurm/current
   libjpeg-turbo/2.1.0                    snappy/1.1.8
   libmd/1.0.3                            sqlite/3.36.0
   libpciaccess/0.16                      squashfs/4.4
   libpng/1.6.37                          subversion/1.14.0
   libpthread-stubs/0.4                   suite-sparse/5.10.1-intel-mkl
   libseccomp/2.3.3                       superlu-mt/3.1-intel-mkl
   libsigsegv/2.13                        superlu/5.3.0-intel-mkl
   libssh2/1.8.0                          swig/4.0.2
   libtiff/4.3.0                          sz/2.1.12
   libtirpc/1.2.6                         tar/1.34
   libtool/2.4.6                          tcl/8.6.11
   libunistring/0.9.10                    tcsh/6.22.02
   libx11/1.7.0                           texinfo/6.5
   libxau/1.0.8                           time/1.9
   libxcb/1.14                            tk/8.6.11
   libxdmcp/1.1.2                         ucx/1.11.0
   libxext/1.3.3                          unzip/6.0
   libxfont/1.5.2                         utf8proc/2.4.0
   libxft/2.3.2                           util-linux-uuid/2.36.2
   libxml2/2.9.12                         util-linux/2.37.2
   libxpm/3.5.12                          util-macros/1.19.3
   libxrender/0.9.10                      voropp/0.4.6
   libxscrnsaver/1.2.2                    xcb-proto/1.14.1
   libyaml/0.2.5                          xextproto/7.3.0
   likwid/5.2.0                           xproto/7.0.31
   llvm/13.0.0                            xtrans/1.3.5
   lua/5.3.5                              xz/5.2.5
   lvm2/2.03.05                           yasm/1.3.0
   lz4/1.9.3                              z3/4.8.9
   m4/1.4.19                              zfp/0.5.5
   magma/2.6.1-intel-mkl                  zip/3.0
   meson/0.60.0                           zlib/1.2.11
   metis/5.1.0                            zstd/1.5.0

Loading modules hierarchicaly

Modules need to be loaded with the right module paths. This system is based on lmod, which is organized hierarchically. This means that the modules you see depend on the ones you have already loaded: for example, if you don't load openmpi you don't see hdf5, etc. To find modules in the hierarchy, you can use the module spider command.

For example, if you are looking for the package trilinos, you can type the following:

[NetID@login01 ~]$ module spider trilinos

  trilinos: trilinos/13.0.1-intel-mkl

    You will need to load all module(s) on any one of the lines below before the "trilinos/13.0.1-intel-mkl" module is available to load.

      2023r1  openmpi/4.1.1

      The Trilinos Project is an effort to develop algorithms and enabling
      technologies within an object-oriented software framework for the
      solution of large-scale, complex multi-physics engineering and
      scientific problems. A unique design feature of Trilinos is its focus on

The module manager tells you that trilinos is available and also tells you exactly which modules you need to load first, in this case 2023r1 and openmpi/4.1.1. As you can see, loading openmpi/4.1.1 opens another sub-set of available modules that depend on OpenMPI to function:

[NetID@login01 ~]$ module load openmpi
[NetID@login01 ~]$ module avail

----------------- /apps/arch/2023r1/lmod/linux-rhel8-x86_64/openmpi/4.1.1-fezcq73/Core -----------------
   adios2/2.7.1                 netcdf-fortran/4.5.3        py-mpi4py/3.1.2
   gromacs/2021.3-intel-mkl     opencoarrays/2.7.1          py-tensorflow/2.4.1-intel-mkl
   hdf5/1.10.7                  openfoam-org/8              py-torch/1.10.0-intel-mkl
   hypre/2.23.0-intel-mkl       openfoam/2106-intel-mkl     scotch/6.1.1
   lammps/20210310-intel-mkl    parallel-netcdf/1.12.2      superlu-dist/7.1.1-intel-mkl
   mumps/5.4.0-intel-mkl        parmetis/4.0.3              trilinos/13.0.1-intel-mkl
   netcdf-c/4.8.1               petsc/3.16.1-intel-mkl      valgrind/3.17.0
   netcdf-cxx4/4.3.1            py-h5py/2.10.0-intel-mkl    wrf/4.2

So, to load trilinos, all you need to do now is the following:

[NetID@login01 ~]$ module load 2023r1
[NetID@login01 ~]$ module load openmpi
[NetID@login01 ~]$ module load trilinos

The module system might load additional dependencies as needed:

[NetID@login01 ~]$ module load trilinos

Autoloading hwloc/2.6.0
Autoloading intel-mkl/2020.4.304
Autoloading mumps/5.4.0-intel-mkl
Autoloading py-mpi4py/3.1.2
Autoloading python/3.8.12
Autoloading py-numpy/1.19.5-intel-mkl
Autoloading py-setuptools/58.2.0
Autoloading suite-sparse/5.10.1-intel-mkl
Autoloading swig/4.0.2

And now the following modules are loaded:

[NetID@login01 ~]$ module list

Currently Loaded Modules:
  1) slurm/current    6) intel-mkl/2020.4.304   11) py-numpy/1.19.5-intel-mkl
  2) DefaultModules   7) mumps/5.4.0-intel-mkl  12) suite-sparse/5.10.1-intel-mkl
  3) 2023r1           8) python/3.8.12          13) swig/4.0.2
  4) openmpi/4.1.1    9) py-mpi4py/3.1.2        14) trilinos/13.0.1-intel-mkl
  5) hwloc/2.6.0     10) py-setuptools/58.2.0

Please note, that slurm/current, and DefaultModules are loaded automatically upon login, and are always available. 2023r1, openmpi/4.1.1, and trilinos/13.0.1-intel-mkl we just loaded ourselves explicitly. The remaining packages were loaded as dependencies together with trilinos.

You can use module unload to unload the loaded module, and module show to display module information.

Environment variables

Finally, most modules from the current software stack set their own environment variable of the format $MODULENAME_ROOT. For example, for trilinos:

[<NetID>@login02 ~]$ echo $TRILINOS_ROOT

If you cd to the above directory, you will find all the usual suspects sub-folders, such as bin, include, and lib.

My module does not seem to set $MODULENAME_ROOT. Help!

Some modules might set different environment variables, if a more "standard" variable exists and is commonly recognized. For example, the cuda/11.6 module sets $CUDA_PATH, not $CUDA_ROOT. Other examples are intel/oneapi-all, and intel-mkl modules that set the $MKLROOT variable.

Luckily, you can always find which variables are set by issuing module show command:

[<NetID>@login01 ~]$ module show cuda/11.6
whatis("CUDA 11.6")
whatis("NVIDIA CUDA Toolkit provides a development environment for creating high performance GPU-accelerated applications.")

Same procedure applies to all modules.

Beta phase: be careful with purging modules!

Please be careful with issuing the module purge and module unuse commands!

Depending on what you do exactly, this might not only remove loaded modules, but also disable the DelftBlue software stack and/or the vendor-preinstalled software stack. The problem with this is that (for the time being, during the beta phase) some modules still rely on both stacks. For example, the slurm module might not work as expected. If this happened, you have to re-activate the standard vendor-preinstalled modules, as well as the DelftBlue stack, again:

Vendor-preinstalled software stack:

$ module use /cm/local/modulefiles 
$ module use /cm/shared/modulefiles

DelftBlue software stack:

$ module use /mnt/shared/apps/modulefiles
Placing your standard modules to .bashrc

TODO - Use the intended functionality for this (using $HOME/.module, see man module for details)

In order to make your life easier, you can place all your typically used modules to the /home/<netid>/.bashrc file. These modules will then be loaded automatically every time you log in. Let's open the .bashrc file with vi:

vi /home/<netid>/.bashrc

This is the default contents of the file, keep this as is:

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc

# User specific environment
if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
export PATH

# Uncomment the following line if you don't like systemctl's auto-paging feature:

# User specific aliases and functions

After the last line, we can add any bash command that we want to be executed upon login. For example, I would like to enable the DelftBlue software stack, load the Intel compilers suite, and then load cmake and some Python modules:

# Load DelftBlue software stack modules:
module load 2023r1

# Load DelftBlue software stack Intel compilers:
module load intel/oneapi-all

# Load DelftBlue software stack Python for compute nodes:
module load python
module load py-numpy
module load py-scipy
module load py-matplotlib

# Load cmake:
module load cmake

Upon the next login, I see the following modules loaded automatically:

[<netid>@login03 ~]$ module list

Currently Loaded Modules:

1) shared                    9) py-numpy/1.19.5-intel-mkl  17) libjpeg-turbo/2.1.0
2) slurm/delftblue/21.08.8  10) py-pybind11/2.6.2          18) zlib/1.2.11
3) DefaultModules           11) py-scipy/1.5.2-intel-mkl   19) py-pillow/8.0.0
4) 2023r1                   12) freetype/2.11.0            20) py-pyparsing/2.4.7
5) intel/oneapi-all         13) libpng/1.6.37              21) py-python-dateutil/2.8.2
6) python/3.8.12            14) py-six/1.15.0              22) qhull/2020.2
7) intel-mkl/2020.4.304     15) py-cycler/0.10.0           23) py-matplotlib/3.4.3-intel-mkl
8) py-setuptools/58.2.0     16) py-kiwisolver/1.3.2        24) cmake/3.21.4

Please note, that shared, slurm/delftblue/21.08.8, and DefaultModules are loaded automatically upon login, and are always available. 2023r1, intel/oneapi-all, python, py-numpy, py-scipy, py-matplotlib, and cmake we just loaded ourselves explicitly. The remaining packages were loaded as dependencies together with requested packages.