Load software modules¶
Note: this page contains some example output from the module command
that may not be up-to-date, it is meant for illustration purposes only.
Default module stack¶
The default DelftBlue software stack is called 2023r1:
[<netid>@login01 ~]$ module avail
-------------------------------------------------------------------- /apps/noarch/modulefiles ------------------------------------------------------------
2022r2 2023r1 2023r1-gcc11 2023r1-intel
The 2023r1 software stack should be used by default. The others are built using specific compilers in case you want to use these
compilers and link against libraries from the module system (e.g., MPI, OpenBLAS or PETSc).
The default stack can be loaded as follows:
Within the software stack, you will see two types of modules:
- modules that can be loaded for all nodes (
/apps/generic/modulefiles); - modules specific to
computeorgpunodes (/apps/arch/2023r1/lmod/linux-rhel8-x86_64/Core)
Different partitions may have different modules!
Modules you see when you execute module avail depend on which partition of DelftBlue you are on. compute and login nodes have their own collection of software, gpu nodes have their own collection of software, and visual nodes have yet another collection of software. For example, do not just assume that exactly the same modules with exactly the same version names are available both on login and gpu nodes!
--------------------------------------- /apps/noarch/modulefiles ---------------------------------------
2023r1 (L)
-------------------------------------- /apps/generic/modulefiles ---------------------------------------
abaqus/2022 feko/2022 miniconda3/4.12.0
ansys/2021R2 intel/oneapi-all_2022.3 mstar/solver_3.7.53
comsol/5.6 intel/oneapi-all mstar/3.7.53 (D)
cuda/11.6 (D) intel/oneapi nvhpc/22.3
cuda/11.7 intel/psxe_2020 (D) paraview/5.9.1 (D)
desktop/vnc ls-dyna/13.0.1 powerflow/6.2021.R6
diana/10.5 matlab/R2021b slurm/current (L,D)
---------------------------- /apps/arch/2023r1/lmod/linux-rhel8-x86_64/Core ----------------------------
alsa-lib/1.2.3.2 mkfontdir/1.0.7
apr-util/1.6.1 mkfontscale/1.1.2
apr/1.6.2 mpc/1.1.0
autoconf-archive/2019.01.06 mpfr/4.1.0
autoconf/2.69 nasm/2.15.05
automake/1.16.3 nccl/2.9.9-1
bazel/3.7.2 ncurses/6.2
bdftopcf/1.0.5 ninja/1.10.2
berkeley-db/18.1.40 numactl/2.0.14
binutils/2.37 openjdk/11.0.12_7
bison/3.8.2 openmpi/4.1.1
boost/1.77.0 openssh/8.7p1
bzip2/1.0.8 openssl/1.1.1l
c-ares/1.15.0 pango/1.42.0
c-blosc/1.21.0 paraview/5.9.1
cairo/1.16.0 pcre/8.44
cgal/4.13 pcre2/10.36
cmake/3.21.4 perl-alien-svn/1.8.11.0
cpio/2.13 perl-data-dumper/2.173
cryptsetup/2.3.5 perl-module-build/0.4224
cuda/10.2.89 perl/5.34.0
cudnn/8.2.0.53-10.2 pixman/0.40.0
curl/7.79.0 pkgconf/1.8.0
diffutils/3.8 pmix/3.2.1
doxygen/1.9.2 popt/1.16
eigen/3.4.0 protobuf/3.15.7
expat/2.4.1 psimd/2020-05-17
ffmpeg/4.3.2 pthreadpool/2021-04-13
findutils/4.8.0 py-absl-py/0.13.0
flatbuffers/1.12.0 py-astunparse/1.6.3
flex/2.6.4 py-certifi/2021.10.8
font-util/1.3.2 py-cppy/1.1.0
fontconfig/2.13.94 py-cycler/0.10.0
fontsproto/2.1.3 py-cython/0.29.24
freetype/2.11.0 py-future/0.18.2
fribidi/1.0.5 py-gast/0.3.3
fxdiv/2020-04-17 py-google-pasta/0.2.0
gawk/5.1.0 py-grpcio/1.32.0
gcc/11.2.0 py-joblib/1.0.1
gdbm/1.19 py-keras-preprocessing/1.1.2-intel-mkl
gettext/0.21 py-kiwisolver/1.3.2
ginkgo/1.4.0 py-matplotlib/3.4.3-intel-mkl
git-lfs/2.11.0 py-numpy/1.19.5-intel-mkl
git/2.31.1 py-opt-einsum/3.3.0-intel-mkl
glib/2.70.0 py-packaging/21.0
gmake/4.3 py-pillow/8.0.0
gmp/6.2.1 py-pip/21.1.2
gnuplot/5.4.2 py-pkgconfig/1.5.1
go-bootstrap/1.4-bootstrap-20171003 py-protobuf/3.17.3
go/1.17.2 py-pybind11/2.6.2
gobject-introspection/1.56.1 py-pyparsing/2.4.7
gperf/3.1 py-python-dateutil/2.8.2
gsl/2.7 py-pyyaml/5.3.1
harfbuzz/2.6.8 py-scikit-learn/1.0.1-intel-mkl
help2man/1.47.16 py-scipy/1.5.2-intel-mkl
hwloc/2.6.0 py-setuptools-scm/6.3.2
icu4c/67.1 py-setuptools/58.2.0
inputproto/2.3.2 py-six/1.15.0
intel-mkl/2020.4.304 py-termcolor/1.1.0
jasper/2.0.32 py-threadpoolctl/2.0.0
json-c/0.15 py-tomli/1.2.1
julia/1.6.3-intel-mkl py-tqdm/4.62.3
kbproto/1.0.7 py-typing-extensions/3.7.4.3
kim-api/2.2.1 py-wheel/0.36.2
krb5/1.19.2 py-wrapt/1.12.1
libaio/0.3.110 python/3.8.12
libbsd/0.11.3 qhull/2020.2
libcerf/1.3 r/4.1.1
libedit/3.1-20210216 rdma-core/54mlnx1
libevent/2.1.8 readline/8.1
libfabric/1.13.2 renderproto/0.11.1
libffi/3.3 rust/1.51.0
libfontenc/1.1.3 scala/2.13.1
libgd/2.2.4 scons/3.1.2
libgit2/1.1.1 scrnsaverproto/1.2.2
libgpg-error/1.42 serf/1.3.9
libiconv/1.16 shadow/4.8.1
libidn2/2.3.0 slurm/current
libjpeg-turbo/2.1.0 snappy/1.1.8
libmd/1.0.3 sqlite/3.36.0
libpciaccess/0.16 squashfs/4.4
libpng/1.6.37 subversion/1.14.0
libpthread-stubs/0.4 suite-sparse/5.10.1-intel-mkl
libseccomp/2.3.3 superlu-mt/3.1-intel-mkl
libsigsegv/2.13 superlu/5.3.0-intel-mkl
libssh2/1.8.0 swig/4.0.2
libtiff/4.3.0 sz/2.1.12
libtirpc/1.2.6 tar/1.34
libtool/2.4.6 tcl/8.6.11
libunistring/0.9.10 tcsh/6.22.02
libx11/1.7.0 texinfo/6.5
libxau/1.0.8 time/1.9
libxcb/1.14 tk/8.6.11
libxdmcp/1.1.2 ucx/1.11.0
libxext/1.3.3 unzip/6.0
libxfont/1.5.2 utf8proc/2.4.0
libxft/2.3.2 util-linux-uuid/2.36.2
libxml2/2.9.12 util-linux/2.37.2
libxpm/3.5.12 util-macros/1.19.3
libxrender/0.9.10 voropp/0.4.6
libxscrnsaver/1.2.2 xcb-proto/1.14.1
libyaml/0.2.5 xextproto/7.3.0
likwid/5.2.0 xproto/7.0.31
llvm/13.0.0 xtrans/1.3.5
lua/5.3.5 xz/5.2.5
lvm2/2.03.05 yasm/1.3.0
lz4/1.9.3 z3/4.8.9
m4/1.4.19 zfp/0.5.5
magma/2.6.1-intel-mkl zip/3.0
meson/0.60.0 zlib/1.2.11
metis/5.1.0 zstd/1.5.0
Loading modules hierarchicaly¶
Modules need to be loaded with the right module paths. This system is based on lmod, which is organized hierarchically. This means that the modules you see depend on the ones you have already loaded: for example, if you don't load openmpi you don't see hdf5, etc. To find modules in the hierarchy, you can use the module spider command.
For example, if you are looking for the package trilinos, you can type the following:
[NetID@login01 ~]$ module spider trilinos
----------------------------------------------------------------------------------------------------
trilinos: trilinos/13.0.1-intel-mkl
----------------------------------------------------------------------------------------------------
You will need to load all module(s) on any one of the lines below before the "trilinos/13.0.1-intel-mkl" module is available to load.
2023r1 openmpi/4.1.1
Help:
The Trilinos Project is an effort to develop algorithms and enabling
technologies within an object-oriented software framework for the
solution of large-scale, complex multi-physics engineering and
scientific problems. A unique design feature of Trilinos is its focus on
packages.
The module manager tells you that trilinos is available and also tells you exactly which modules you need to load first, in this case 2023r1 and openmpi/4.1.1. As you can see, loading openmpi/4.1.1 opens another sub-set of available modules that depend on OpenMPI to function:
[NetID@login01 ~]$ module load openmpi
[NetID@login01 ~]$ module avail
----------------- /apps/arch/2023r1/lmod/linux-rhel8-x86_64/openmpi/4.1.1-fezcq73/Core -----------------
adios2/2.7.1 netcdf-fortran/4.5.3 py-mpi4py/3.1.2
gromacs/2021.3-intel-mkl opencoarrays/2.7.1 py-tensorflow/2.4.1-intel-mkl
hdf5/1.10.7 openfoam-org/8 py-torch/1.10.0-intel-mkl
hypre/2.23.0-intel-mkl openfoam/2106-intel-mkl scotch/6.1.1
lammps/20210310-intel-mkl parallel-netcdf/1.12.2 superlu-dist/7.1.1-intel-mkl
mumps/5.4.0-intel-mkl parmetis/4.0.3 trilinos/13.0.1-intel-mkl
netcdf-c/4.8.1 petsc/3.16.1-intel-mkl valgrind/3.17.0
netcdf-cxx4/4.3.1 py-h5py/2.10.0-intel-mkl wrf/4.2
So, to load trilinos, all you need to do now is the following:
[NetID@login01 ~]$ module load 2023r1
[NetID@login01 ~]$ module load openmpi
[NetID@login01 ~]$ module load trilinos
The module system might load additional dependencies as needed:
[NetID@login01 ~]$ module load trilinos
Autoloading hwloc/2.6.0
Autoloading intel-mkl/2020.4.304
Autoloading mumps/5.4.0-intel-mkl
Autoloading py-mpi4py/3.1.2
Autoloading python/3.8.12
Autoloading py-numpy/1.19.5-intel-mkl
Autoloading py-setuptools/58.2.0
Autoloading suite-sparse/5.10.1-intel-mkl
Autoloading swig/4.0.2
And now the following modules are loaded:
[NetID@login01 ~]$ module list
Currently Loaded Modules:
1) slurm/current 6) intel-mkl/2020.4.304 11) py-numpy/1.19.5-intel-mkl
2) DefaultModules 7) mumps/5.4.0-intel-mkl 12) suite-sparse/5.10.1-intel-mkl
3) 2023r1 8) python/3.8.12 13) swig/4.0.2
4) openmpi/4.1.1 9) py-mpi4py/3.1.2 14) trilinos/13.0.1-intel-mkl
5) hwloc/2.6.0 10) py-setuptools/58.2.0
Please note, that slurm/current, and DefaultModules are loaded automatically upon login, and are always available. 2023r1, openmpi/4.1.1, and trilinos/13.0.1-intel-mkl we just loaded ourselves explicitly. The remaining packages were loaded as dependencies together with trilinos.
You can use module unload to unload the loaded module, and module show to display module information.
Environment variables¶
Finally, most modules from the current software stack set their own environment variable of the format $MODULENAME_ROOT. For example, for trilinos:
[<NetID>@login02 ~]$ echo $TRILINOS_ROOT
/apps/arch/2023r1/software/linux-rhel8-skylake_avx512/gcc-8.5.0/trilinos-13.0.1-whcmjc5i7xbqwu7quorptx4mtwsvdyvt
If you cd to the above directory, you will find all the usual suspects sub-folders, such as bin, include, and lib.
My module does not seem to set $MODULENAME_ROOT. Help!
Some modules might set different environment variables, if a more "standard" variable exists and is commonly recognized. For example, the cuda/11.6 module sets $CUDA_PATH, not $CUDA_ROOT. Other examples are intel/oneapi-all, and intel-mkl modules that set the $MKLROOT variable.
Luckily, you can always find which variables are set by issuing module show command:
[<NetID>@login01 ~]$ module show cuda/11.6
----------------------------------------------------------------------------------------------------
/apps/generic/modulefiles/cuda/11.6.lua:
----------------------------------------------------------------------------------------------------
conflict("cuda")
whatis("CUDA 11.6")
whatis("NVIDIA CUDA Toolkit provides a development environment for creating high performance GPU-accelerated applications.")
setenv("CUDA_PATH","/beegfs/apps/generic/cuda-11.6")
prepend_path("PATH","/beegfs/apps/generic/cuda-11.6/bin")
prepend_path("CPATH","/beegfs/apps/generic/cuda-11.6/include")
prepend_path("LIBRARY_PATH","/beegfs/apps/generic/cuda-11.6/lib64")
prepend_path("LD_LIBRARY_PATH","/beegfs/apps/generic/cuda-11.6/lib64")
Same procedure applies to all modules.
Beta phase: be careful with purging modules!
Please be careful with issuing the module purge and module unuse commands!
Depending on what you do exactly, this might not only remove loaded modules, but also disable the DelftBlue software stack and/or the vendor-preinstalled software stack. The problem with this is that (for the time being, during the beta phase) some modules still rely on both stacks. For example, the slurm module might not work as expected. If this happened, you have to re-activate the standard vendor-preinstalled modules, as well as the DelftBlue stack, again:
Vendor-preinstalled software stack:
DelftBlue software stack:
Placing your standard modules to .bashrc
TODO - Use the intended functionality for this (using $HOME/.module, see man module for details)
In order to make your life easier, you can place all your typically used modules to the /home/<netid>/.bashrc file. These modules will then be loaded automatically every time you log in. Let's open the .bashrc file with vi:
This is the default contents of the file, keep this as is:
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# User specific environment
if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
then
PATH="$HOME/.local/bin:$HOME/bin:$PATH"
fi
export PATH
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions
After the last line, we can add any bash command that we want to be executed upon login. For example, I would like to enable the DelftBlue software stack, load the Intel compilers suite, and then load cmake and some Python modules:
# Load DelftBlue software stack modules:
module load 2023r1
# Load DelftBlue software stack Intel compilers:
module load intel/oneapi-all
# Load DelftBlue software stack Python for compute nodes:
module load python
module load py-numpy
module load py-scipy
module load py-matplotlib
# Load cmake:
module load cmake
Upon the next login, I see the following modules loaded automatically:
[<netid>@login03 ~]$ module list
Currently Loaded Modules:
1) shared 9) py-numpy/1.19.5-intel-mkl 17) libjpeg-turbo/2.1.0
2) slurm/delftblue/21.08.8 10) py-pybind11/2.6.2 18) zlib/1.2.11
3) DefaultModules 11) py-scipy/1.5.2-intel-mkl 19) py-pillow/8.0.0
4) 2023r1 12) freetype/2.11.0 20) py-pyparsing/2.4.7
5) intel/oneapi-all 13) libpng/1.6.37 21) py-python-dateutil/2.8.2
6) python/3.8.12 14) py-six/1.15.0 22) qhull/2020.2
7) intel-mkl/2020.4.304 15) py-cycler/0.10.0 23) py-matplotlib/3.4.3-intel-mkl
8) py-setuptools/58.2.0 16) py-kiwisolver/1.3.2 24) cmake/3.21.4
Please note, that shared, slurm/delftblue/21.08.8, and DefaultModules are loaded automatically upon login, and are always available. 2023r1, intel/oneapi-all, python, py-numpy, py-scipy, py-matplotlib, and cmake we just loaded ourselves explicitly. The remaining packages were loaded as dependencies together with requested packages.