Installing software with Spack¶
HPC software can have multiple dependencies and be quite challenging to install. While Conda is a popular tool for general-purpose software, HPC software often runs close to the hardware and builds directly on several layers of libraries and drivers that need to work optimally together. In particular if your application uses MPI and/or CUDA, it is worth checking if it can be installed via Spack (the Supercomputer PACKage manager). This has the distinctive advantage over conda that spack will allow you to build your custom packages and modules on top of (and thus fully compatible with) the DelftBlue software stack.
This is only a brief walkthrough specific to DelftBlue. For extensive documentation and tutorials, see the Spack documentation.
Note
It is crucial that you do not install certain dependencies. We use an upstream construct below to avoid this. Typical libraries that will likely render your software unusable are SLURM, MPI, and possibly CUDA.
Where should I install software?¶
By default, software will be installed in a location within the spack directory that you get after cloning the spack repository (see below).
It is recommended to install software packages in your /home folder.
If the size of the installation is very large, you may instead use /scratch temporarily
(note that files get cleaned up after 6 months unless you update their access time stamp).
If you want to install software to be used by multiple people (e.g., a course or a research group), please request a project folder for this purpose.
Clone the Spack repository and activate spack¶
First load the current software stack and Python, and get the tool itself:
Note that we specify a release version to use: The default branch develop of spack can be quite unstable and lead to
unexpected errors. Also note that this is not the latest release: It is the one used to build our software stack, and
we highly recommend using it. Not having the latest version of spack does not mean you cannot install the latest version
of your software.
Now activate the spack command by sourcing a specific file. Assuming that you cloned spack into your home directory:
You may want to put this line into your${HOME}/.bashrc file so that you always have access to spack,
or into some other file like env-spack.sh that you can source when needed.
After the above line, the command spack should be available, and $SPACK_ROOT points to its base directory.
Copy required configuration files¶
We want to build additional software "on top" of the existing software stacks on DelftBlue in order to avoid
multiple installations of the same package by users (to some extent). Also, we want to create module files for your
own software that behave similar to the existing ones. To this end, you
should copy some configuration files to your own spack directory.
E.g., for the 2024r spack, we placed them in /projects/unsupported/ for you:
Find compilers and check if dependencies are found¶
To make spack look for usable compilers, type the following command. Make sure to load the software stack on top of which you want to build (the "upstream stack", here 2024r1), because our stacks are typically not built with the system default compiler.
The output looks similar to this:==> Added 2 new compilers to /home/<netid>/.spack/linux/compilers.yaml
gcc@8.5.0 gcc@11.3.0
==> Compilers are defined in the following files:
/home/<netid>/.spack/linux/compilers.yaml
You can now check if important dependencies like MPI are found correctly:
The output includes the options used to build the package found, and a hash at the start of the line that uniquely identifies the installed package. This hash can be used to force spack to pick a particular dependency later on:-- linux-rhel8-cascadelake / gcc@11.3.0 -------------------------
w6w5qi5 openmpi@4.1.6~atomics~cuda+cxx~cxx_exceptions~gpfs~internal-hwloc~internal-pmix~java+legacylaunchers~lustre~memchecker~openshmem~orterunprefix+pmi+romio+rsh~singularity+static+vt+wrapper-rpath build_system=autotools fabrics=ucx schedulers=slurm
==> 1 installed package
Finding and configuring your software¶
As an example, let's try to install the Finite Element package `FEniCS. To list related packages, use:
which reveals a variety of options:fenics fenics-dolfinx py-fenics-basix py-fenics-dolfinx py-fenics-ffcx py-fenics-instant py-fenics-ufl-legacy
fenics-basix fenics-ufcx py-fenics-dijitso py-fenics-ffc py-fenics-fiat py-fenics-ufl
fenics and fenics-basix, so we will use it.
The output also provides a short description, available versions and configuration options (called "variants" in spack).
Let's see what requesting an installation of fenics-dolfinx would amount to. To make it a bit more interesting, we enable
the option +slepc that pulls in an external eigensolver library.
Note
The first time you type this command, you may experience spack bootstrapping, so type it again to get only the relevant output. Pipe the output into a file for easier inspection.
- With the
-Iflag we request to see which packages are available already and which would have to be installed. - The
--reuseflag is highly recommended: It tells spack to try to use packages that are already installed as much as possible. - The plus (
+) in front of a variant enables it, whereas a tilde (~) would disable it.
The complete output looks like this at the time of writing:
Input spec
--------------------------------
- fenics-dolfinx+slepc
Concretized
--------------------------------
- fenics-dolfinx@0.6.0%gcc@11.3.0~adios2~ipo+slepc build_system=cmake build_type=Release generator=make partitioners=parmetis arch=linux-rhel8-cascadelake
[^] ^boost@1.83.0%gcc@11.3.0+atomic+chrono~clanglibcpp~container~context~contract~coroutine+date_time~debug+exception~fiber+filesystem+graph~graph_parallel~icu+iostreams~json+locale+log+math~mpi+multithreaded~nowide~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded~stacktrace+system~taggedlayout+test+thread+timer~type_erasure~versionedlayout+wave build_system=generic cxxstd=11 patches=a440f96 visibility=hidden arch=linux-rhel8-cascadelake
[^] ^bzip2@1.0.8%gcc@11.3.0~debug~pic+shared build_system=generic arch=linux-rhel8-cascadelake
[^] ^xz@5.4.1%gcc@11.3.0~pic build_system=autotools libs=shared,static arch=linux-rhel8-cascadelake
[^] ^zlib@1.2.13%gcc@11.3.0+optimize+pic+shared build_system=makefile arch=linux-rhel8-cascadelake
[^] ^zstd@1.5.5%gcc@11.3.0+programs build_system=makefile compression=none libs=shared,static arch=linux-rhel8-cascadelake
[^] ^cmake@3.27.7%gcc@11.3.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rhel8-cascadelake
[^] ^curl@8.4.0%gcc@11.3.0~gssapi~ldap+libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rhel8-cascadelake
[^] ^libidn2@2.3.4%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^libunistring@1.1%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^nghttp2@1.57.0%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^openssl@3.1.3%gcc@11.3.0~docs+shared build_system=generic certs=mozilla arch=linux-rhel8-cascadelake
[^] ^ca-certificates-mozilla@2023-05-30%gcc@11.3.0 build_system=generic arch=linux-rhel8-cascadelake
[^] ^ncurses@6.4%gcc@11.3.0~symlinks+termlib abi=none build_system=autotools arch=linux-rhel8-cascadelake
- ^fenics-basix@0.6.0%gcc@11.3.0~ipo build_system=cmake build_type=Release generator=make arch=linux-rhel8-cascadelake
[^] ^openblas@0.3.24%gcc@11.3.0~bignuma~consistent_fpcsr+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=openmp arch=linux-rhel8-cascadelake
- ^fenics-ufcx@0.6.0%gcc@11.3.0~ipo build_system=cmake build_type=Release generator=make arch=linux-rhel8-cascadelake
[^] ^gmake@4.4.1%gcc@11.3.0~guile build_system=generic arch=linux-rhel8-cascadelake
[^] ^hdf5@1.14.3%gcc@11.3.0+cxx+fortran+hl~ipo~java~map+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=Release generator=make arch=linux-rhel8-cascadelake
[^] ^openmpi@4.1.6%gcc@11.3.0~atomics~cuda+cxx~cxx_exceptions~gpfs~internal-hwloc~internal-pmix~java+legacylaunchers~lustre~memchecker~openshmem~orterunprefix+pmi+romio+rsh~singularity+static+vt+wrapper-rpath build_system=autotools fabrics=ucx schedulers=slurm arch=linux-rhel8-cascadelake
[^] ^hwloc@2.9.1%gcc@11.3.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-rhel8-cascadelake
[^] ^libpciaccess@0.17%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^util-macros@1.19.3%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^libxml2@2.10.3%gcc@11.3.0+pic~python+shared build_system=autotools arch=linux-rhel8-cascadelake
[^] ^numactl@2.0.14%gcc@11.3.0 build_system=autotools patches=4e1d78c,62fc8a8,ff37630 arch=linux-rhel8-cascadelake
[^] ^autoconf@2.69%gcc@11.3.0 build_system=autotools patches=35c4492,7793209,a49dd5b arch=linux-rhel8-cascadelake
[^] ^automake@1.16.5%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^libtool@2.4.7%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^m4@1.4.19%gcc@11.3.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-rhel8-cascadelake
[^] ^libsigsegv@2.14%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^openssh@9.5p1%gcc@11.3.0+gssapi build_system=autotools arch=linux-rhel8-cascadelake
[^] ^krb5@1.20.1%gcc@11.3.0+shared build_system=autotools arch=linux-rhel8-cascadelake
[^] ^findutils@4.9.0%gcc@11.3.0 build_system=autotools patches=440b954 arch=linux-rhel8-cascadelake
[^] ^libedit@3.1-20210216%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^libxcrypt@4.4.35%gcc@11.3.0~obsolete_api build_system=autotools patches=4885da3 arch=linux-rhel8-cascadelake
[^] ^perl@5.38.0%gcc@11.3.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-rhel8-cascadelake
[^] ^berkeley-db@18.1.40%gcc@11.3.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rhel8-cascadelake
[^] ^gdbm@1.23%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^pmix@5.0.1%gcc@11.3.0~docs+pmi_backwards_compatibility~python~restful build_system=autotools arch=linux-rhel8-cascadelake
[^] ^libevent@2.1.12%gcc@11.3.0+openssl build_system=autotools arch=linux-rhel8-cascadelake
[e] ^slurm@21.08.8%gcc@11.3.0~cgroup~gtk~hdf5~hwloc~mariadb~nvml~pam~pmix+readline~restd~rsmi build_system=autotools sysconfdir=PREFIX/etc arch=linux-rhel8-cascadelake
[e] ^ucx@1.15.0%gcc@11.3.0~assertions~backtrace_detail~cma~cuda~dc~debug~dm+examples~gdrcopy~gtest~ib_hw_tm~java~knem~logging~mlx5_dv+openmp+optimizations~parameter_checking+pic~rc~rdmacm~rocm+thread_multiple~ucg~ud~verbs~vfs~xpmem build_system=autotools libs=shared,static opt=3 simd=auto arch=linux-rhel8-cascadelake
[^] ^parmetis@4.0.3%gcc@11.3.0~gdb~int64~ipo+shared build_system=cmake build_type=Release generator=make patches=4f89253,50ed208,704b84f arch=linux-rhel8-cascadelake
[^] ^metis@5.1.0%gcc@11.3.0~gdb~int64~ipo~real64+shared build_system=cmake build_type=Release generator=make patches=4991da9,93a7903,b1225da arch=linux-rhel8-cascadelake
[^] ^petsc@3.20.1%gcc@11.3.0~X~batch~cgns~complex~cuda~debug+double~exodusii~fftw+fortran~giflib~hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~kokkos~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi+mumps+openmp~p4est~parmmg+ptscotch~random123~rocm~saws~scalapack+shared~strumpack~suite-sparse+superlu-dist~sycl~tetgen~trilinos~valgrind build_system=generic clanguage=C memalign=none arch=linux-rhel8-cascadelake
[^] ^diffutils@3.9%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^libiconv@1.17%gcc@11.3.0 build_system=autotools libs=shared,static arch=linux-rhel8-cascadelake
[^] ^hypre@2.29.0%gcc@11.3.0~caliper~complex~cuda~debug+fortran~gptune~int64~internal-superlu~magma~mixedint+mpi~openmp~rocm+shared~superlu-dist~sycl~umpire~unified-memory build_system=autotools arch=linux-rhel8-cascadelake
[^] ^mumps@5.5.1%gcc@11.3.0~blr_mt+complex+double+float~incfort~int64+metis+mpi+openmp+parmetis+ptscotch~scotch+shared build_system=generic patches=373d736 arch=linux-rhel8-cascadelake
[^] ^netlib-scalapack@2.2.0%gcc@11.3.0~ipo~pic+shared build_system=cmake build_type=Release generator=make patches=072b006,1c9ce5f,244a9aa arch=linux-rhel8-cascadelake
[^] ^python@3.10.12%gcc@11.3.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,ebdca64,f2fd060 arch=linux-rhel8-cascadelake
[^] ^expat@2.5.0%gcc@11.3.0+libbsd build_system=autotools arch=linux-rhel8-cascadelake
[^] ^libbsd@0.11.7%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^libmd@1.0.4%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^gettext@0.22.3%gcc@11.3.0+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-rhel8-cascadelake
[^] ^tar@1.34%gcc@11.3.0 build_system=autotools zip=pigz arch=linux-rhel8-cascadelake
[^] ^pigz@2.7%gcc@11.3.0 build_system=makefile arch=linux-rhel8-cascadelake
[^] ^libffi@3.4.4%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^readline@8.2%gcc@11.3.0 build_system=autotools patches=bbf97f1 arch=linux-rhel8-cascadelake
[^] ^sqlite@3.43.2%gcc@11.3.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-rhel8-cascadelake
[^] ^util-linux-uuid@2.38.1%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^scotch@7.0.4%gcc@11.3.0+compression+esmumps~int64~ipo~metis+mpi~mpi_thread+shared+threads build_system=cmake build_type=Release generator=make arch=linux-rhel8-cascadelake
[^] ^bison@3.8.2%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^flex@2.6.4%gcc@11.3.0+lex~nls build_system=autotools patches=f8b85a0 arch=linux-rhel8-cascadelake
[^] ^help2man@1.49.3%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^superlu-dist@8.1.2%gcc@11.3.0~cuda~int64~ipo+openmp~rocm+shared build_system=cmake build_type=Release generator=make arch=linux-rhel8-cascadelake
[^] ^pkgconf@1.9.5%gcc@11.3.0 build_system=autotools arch=linux-rhel8-cascadelake
[^] ^pugixml@1.13%gcc@11.3.0~ipo+pic+shared build_system=cmake build_type=Release generator=make arch=linux-rhel8-cascadelake
- ^slepc@3.20.0%gcc@11.3.0+arpack~blopex~cuda~hpddm~rocm build_system=generic arch=linux-rhel8-cascadelake
- ^arpack-ng@3.9.0%gcc@11.3.0~icb~ipo+mpi+shared build_system=cmake build_type=Release generator=make arch=linux-rhel8-cascadelake
The result is a long list of packages, starting with fenics-dolfinx on line 1.
Every package has a version and a list of variants, "~" means disabled, "+" means enabled as before.
The first column contains a minus sign (-) for packages that would have to be installed, and a
hat ([^] for packages that are already available either in your own spack or the upstream (i.e., the DelftBlue software stack).
An [e] means that the dependency is externally installed. Our config file does not allow spack to install it to avoid faulty
installations that, e.g., do not work together with our SLURM job manager.
Installing the software¶
From the output it is clear that spack figured out an excellent solution for us: All dependencies are already available except for the three FEniCS packages, the direct dependency SLEPc that we requested, and another one (Arpack) that SLEPc pulls in in turn. We can now install all of these with a single command, and produce nice module files for us to use in our daily work:
The line is identical to the spec command above, but we will allow using up to 8 proceses to speed up the compilation and installation process.
If all goes well, the last few lines of output will be similar to this:
==> Installing fenics-dolfinx-0.6.0-depn3m3hnl4ui7ydbqt4fap4uzx45jqp [56/56]
==> No binary for fenics-dolfinx-0.6.0-depn3m3hnl4ui7ydbqt4fap4uzx45jqp found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/eb/eb8ac2bb2f032b0d393977993e1ab6b4101a84d54023a67206e3eac1a8d79b80.tar.gz
==> No patches needed for fenics-dolfinx
==> fenics-dolfinx: Executing phase: 'cmake'
==> fenics-dolfinx: Executing phase: 'build'
==> fenics-dolfinx: Executing phase: 'install'
==> fenics-dolfinx: Successfully installed fenics-dolfinx-0.6.0-depn3m3hnl4ui7ydbqt4fap4uzx45jqp
Stage: 1.38s. Cmake: 11.89s. Build: 17.68s. Install: 0.67s. Post-install: 0.67s. Total: 33.95s
[+] /home/<netid>/spack2024/opt/spack/linux-rhel8-cascadelake/gcc-11.3.0/fenics-dolfinx-0.6.0-depn3m3hnl4ui7ydbqt4fap4uzx45jqp
Using the installed software module¶
We want to have the software we just installed seamlessly integrated with the DelftBlue software. To achieve this,
we can extend the module search path to include our own installation tree.
We used the gcc@11.3.0 compiler above, which is the "Core" compiler configured in spack.
We an find the module paths needed like this:
/home/<netid>/spack2024/share/spack/lmod/linux-rhel8-x86_64/Core
/home/<netid>/spack2024/share/spack/lmod/linux-rhel8-x86_64/openmpi/4.1.6-w6w5qi5/Core
lmod uses hierarchical module files for pakages depending on MPI.
You can execute module use for each path to add it. Put the commands in your .bashrc file in order
to make them available whenever you log in:
module use /home/<netid>/spack2024/share/spack/lmod/linux-rhel8-x86_64/Core
module use /home/<netid>/spack2024/share/spack/lmod/linux-rhel8-x86_64/openmpi/4.1.6-w6w5qi5/Core
Important Notes & Troubleshooting¶
Different node types and GPUs¶
Depending on the node type (e.g., compute-p1, compute-p2, gpu-v100, gpu-a100),
the command module load 2024r1 actually may load a different software stack. This is because
CPUs may have different architectures, and HPC software is typically compiled with high optimization level
so that it may not be executable on different CPUs/GPUs.
If you want to build software for another node type, check the $MODULEPATH environment variable
within a job on that partition, and adjust the configuration file `$SPACK_ROOT/etc/spack/upstream.yaml``
accordingly. Then install your own software within a job running on that partition.
Concretization phase¶
Sometimes the spack spec -I --reuse command does not give the concretization you wanted.
For example, there may be multiple versions of a dependency, and it just doesn't pick the one you want.
To force spack to pick a specific package, determine it's hash by spack find -v -l <dependency>, and pass it
to the concretizer as in spack spec -I --reuse <new package> ^<dependency>/<hash. This may fail if the combination is
not possible, but at least you will get a concrete error message.
Installation phase¶
One common cause of failure during installation is that you run out of space/quota (You should get a clear error message about this). If this is not the issue, you may try to understand where things go wrong and adapt your choice of packages and variants to circumvent it. The official route to resolve installation problems with spack packages is to create an issue on github.