Firedrake¶
Working example (2023r1 software stack)¶
Install some missing Python stuff locally¶
We are using the Python 3.8 module, but no pip module! The following Python packages need to be installed locally:
### Installing Firedrake on DelftBlue
#
# Everything is happening in /scratch/$USER/tools
#
###
### Step 1. Install additional Python packages locally ###
module load 2023r1
module load cmake
module load python
module load openmpi
module load git
module load zlib
module load boost
python -m pip install --user --upgrade pip setuptools wheel virtualenv requests numpy scipy h5py mpi4py six cython mpmath sympy PuLP networkx islpy 'pytest==6.2.5'
Important
Only load the python module, nothing else! In order to install your own pip you might need:
First, load the existing pip module:
Second, update pip:
Third, unload the pip module:
Then, with your own pip, install everything else:
Important
pytest should be exactly version 6.2.5. If you have a newer version installed, remove it via:
And reinstall the exact version with:
As a result, your setup should be:
[<NetID>@login03 ~]$ module list
Currently Loaded Modules:
1) slurm/current 9) gettext/0.21.1 17) libx11/1.7.0 25) libxscrnsaver/1.2.2
2) DefaultModules 10) libiconv/1.16 18) font-util/1.3.2 26) tcl/8.6.12
3) 2023r1 11) libidn2/2.3.0 19) freetype/2.11.1 27) tk/8.6.11
4) cmake/3.24.3 12) openssh/9.1p1 20) libxml2/2.10.1 28) git/2.38.1
5) python/3.9.8 13) openssl/1.1.1s 21) util-linux-uuid/2.38.1 29) zlib/1.2.13
6) openmpi/4.1.4 14) pcre2/10.39 22) fontconfig/2.13.94 30) boost/1.80.0
7) curl/7.85.0 15) perl/5.36.0 23) libxrender/0.9.10
8) expat/2.4.8 16) perl-alien-svn/1.8.11.0 24) libxft/2.3.2
My local pip setup looked like this:
[dpalagin@login03 ~]$ python -m pip list
Package Version
------------------ ---------
ase 3.22.1
attrs 21.4.0
certifi 2021.10.8
charset-normalizer 2.0.12
cycler 0.11.0
Cython 0.29.28
distlib 0.3.4
execnet 1.9.0
filelock 3.6.0
fonttools 4.31.2
h5py 3.6.0
idna 3.3
iniconfig 1.1.1
islpy 2022.1.2
kiwisolver 1.4.2
matplotlib 3.5.1
mpi4py 3.1.3
mpmath 1.2.1
networkx 2.8
numpy 1.22.3
packaging 21.3
Pillow 9.1.0
pip 22.0.4
platformdirs 2.5.2
pluggy 1.0.0
PuLP 2.6.0
py 1.11.0
pyparsing 3.0.7
pytest 6.2.5
pytest-forked 1.4.0
pytest-mock 3.7.0
pytest-xdist 2.5.0
python-dateutil 2.8.2
PyYAML 6.0
requests 2.27.1
scipy 1.8.0
setuptools 62.1.0
six 1.16.0
sympy 1.10.1
toml 0.10.2
tomli 2.0.1
urllib3 1.26.9
virtualenv 20.14.1
wheel 0.37.1
Install PETSc¶
We install PETSc with existing OpenMPI module, and existing Python 3.8 module. We use srun in place of mpiexec. Everything else will be installed by PETSc itself.
### Step 2. Install PETSc ###
export COPTFLAGS="-O3 -march=native -mtune=native"
export CXXOPTFLAGS="-O3 -march=native -mtune=native"
export FOPTFLAGS="-O3 -march=native -mtune=native"
cd /scratch/$USER
mkdir tools
cd tools
mkdir petsc-my-install
curl -OL https://github.com/firedrakeproject/petsc/archive/refs/heads/firedrake.zip
unzip firedrake.zip
mkdir petsc-my-install
cd petsc-firedrake
export PETSC_INSTALL_OPTIONS=" --prefix=/scratch/$USER/tools/petsc-my-install \
--download-pastix \
--download-netcdf \
--download-pnetcdf \
--download-zlib \
--download-chaco \
--download-eigen \
--download-hdf5 \
--download-hypre \
--download-ml \
--download-mumps \
--download-ptscotch \
--download-scalapack \
--download-superlu_dist \
--download-suitesparse \
--download-metis \
--download-parmetis \
--download-hwloc \
--with-c2html=0 \
--with-cxx-dialect=C++11 \
--with-debugging=0 \
--with-fortran-bindings=0 \
--with-shared-libraries=1 \
--with-mpi=1 \
--with-mpi-dir=/apps/arch/2023r1/software/linux-rhel8-skylake_avx512/gcc-8.5.0/openmpi-4.1.4-6ckg3bnys5rinech3zvxnos54hfqnj3g/ \
--with-petsc4py=1 \
--download-fblaslapack \
--with-mpiexec=/cm/shared/apps/slurm/current/bin/srun \
--COPTFLAGS=$COPTFLAGS \
--CXXOPTFLAGS=$CXXOPTFLAGS \
--FOPTFLAGS=$FOPTFLAGS"
./configure $PETSC_INSTALL_OPTIONS
make PETSC_DIR=/scratch/$USER/tools/petsc-firedrake PETSC_ARCH=arch-linux-c-opt all
make PETSC_DIR=/scratch/$USER/tools/petsc-firedrake PETSC_ARCH=arch-linux-c-opt check
make PETSC_DIR=/scratch/$USER/tools/petsc-firedrake PETSC_ARCH=arch-linux-c-opt install
This will generate the following install log for PETSc:
*** Installing petsc4py ***
running install
running build
running build_src
running build_py
running build_ext
writing build/lib.linux-x86_64-3.8/petsc4py/lib/petsc.cfg
running install_lib
creating /mnt/scratch/dpalagin/firedrake-via-job/petsc-my-install/lib/petsc4py
creating /mnt/scratch/dpalagin/firedrake-via-job/petsc-my-install/lib/petsc4py/lib
creating /mnt/scratch/dpalagin/firedrake-via-job/petsc-my-install/lib/petsc4py/include
creating /mnt/scratch/dpalagin/firedrake-via-job/petsc-my-install/lib/petsc4py/include/petsc4py
byte-compiling /mnt/scratch/dpalagin/firedrake-via-job/petsc-my-install/lib/petsc4py/__init__.py to __init__.cpython-38.pyc
byte-compiling /mnt/scratch/dpalagin/firedrake-via-job/petsc-my-install/lib/petsc4py/lib/__init__.py to __init__.cpython-38.pyc
byte-compiling /mnt/scratch/dpalagin/firedrake-via-job/petsc-my-install/lib/petsc4py/__main__.py to __main__.cpython-38.pyc
byte-compiling /mnt/scratch/dpalagin/firedrake-via-job/petsc-my-install/lib/petsc4py/PETSc.py to PETSc.cpython-38.pyc
running install_egg_info
running egg_info
creating petsc4py.egg-info
writing manifest file 'petsc4py.egg-info/SOURCES.txt'
writing manifest file 'petsc4py.egg-info/SOURCES.txt'
Copying petsc4py.egg-info to /mnt/scratch/dpalagin/firedrake-via-job/petsc-my-install/lib/petsc4py-3.16.4-py3.8.egg-info
running install_scripts
=====================================
To use petsc4py, add /mnt/scratch/dpalagin/firedrake-via-job/petsc-my-install/lib to PYTHONPATH
=====================================
gmake[2]: Nothing to be done for 'libmesh-install'.
gmake[2]: Nothing to be done for 'mfem-install'.
gmake[2]: Nothing to be done for 'slepc-install'.
gmake[2]: Nothing to be done for 'hpddm-install'.
gmake[2]: Nothing to be done for 'amrex-install'.
gmake[2]: Nothing to be done for 'bamg-install'.
Install Firedrake¶
### Step 3. Install Firedrake ###
cd /scratch/$USER/tools/
curl -OL https://raw.githubusercontent.com/firedrakeproject/firedrake/master/scripts/firedrake-install
export VENV_NAME=firedrake
export PETSC_DIR=/scratch/$USER/tools/petsc-my-install
export HDF5_DIR=/scratch/$USER/tools/petsc-my-install
export PYTHONPATH=$PYTHONPATH:/scratch/$USER/tools/petsc-my-install/lib
export MPI_BIN_DIR=/apps/arch/2023r1/software/linux-rhel8-skylake_avx512/gcc-8.5.0/openmpi-4.1.4-6ckg3bnys5rinech3zvxnos54hfqnj3g
# Firedrake will look for petsc4py in the source PETSc folder, not in the
# install PETSc folder. We will make a link to circumvent this:
cd /scratch/dpalagin/tools/petsc-my-install
ln -s /scratch/$USER/tools/petsc-firedrake/src src
cd /scratch/$USER/tools/
export FIREDRAKE_INSTALL_OPTIONS="--honour-petsc-dir \
--honour-pythonpath \
--mpicc $MPI_BIN_DIR/bin/mpicc \
--mpicxx $MPI_BIN_DIR/bin/mpicxx \
--mpif90 $MPI_BIN_DIR/bin/mpif90 \
--mpiexec /cm/shared/apps/slurm/current/bin/srun \
--with-blas $PETSC_DIR/lib \
--no-package-manager \
--disable-ssh \
--verbose \
--install thetis \
--venv-name $VENV_NAME"
# Here, I already use modified firedrake-install scirpt, with lines 1051-1062 commented out.
python firedrake-install $FIREDRAKE_INSTALL_OPTIONS
This will generate the following install log for Firedrake:
Installing loopy/
Installing COFFEE/
Installing petsc4py/
Copying petsc4py to virtualenv to install
Installing ufl/
Installing fiat/
Installing FInAT/
Installing tsfc/
Installing PyOP2/
Installing pyadjoint/
Installing firedrake/
Installing thetis/
Successfully installed Firedrake.
Firedrake has been installed in a python venv. You activate it with:
. /mnt/scratch/dpalagin/firedrake/bin/activate
The venv can be deactivated by running:
deactivate
To upgrade Firedrake activate the venv and run firedrake-update
Install log saved in /mnt/scratch/dpalagin/firedrake-install.log
Run the test¶
First, export the right PETSc directory:
Second, activate the firedrake environment:
Finally, run the test:
Output:
[<dpalagin>@login04 firedrake-from-scratch]$ tail -f slurm-6238.out
Element family: dg-dg, degree: 1
2D cell type: triangle
2D mesh: 78 vertices, 100 elements
Horizontal element size: 894.43 ... 894.43 m
Number of 2D elevation DOFs: 300
Number of 2D velocity DOFs: 600
Number of cores: 8
Elevation DOFs per core: ~37.0
dt = 50.0
Using time integrator: CrankNicolson
exp iter time eta norm u norm Tcpu
0 0 0.00 6251.2574 0.0000 0.00
1 2 100.00 5905.5523 1396.2703 147.27
2 4 200.00 5193.8439 2377.4370 0.23
3 6 300.00 4655.3184 2856.8621 0.14
4 8 400.00 4443.5225 3012.5682 0.33
5 10 500.00 4397.6035 3043.0792 0.29
6 12 600.00 4391.3540 3045.5444 0.27
7 14 700.00 4389.7487 3044.7540 0.10
8 16 800.00 4388.7082 3043.5412 0.09
9 18 900.00 4392.2554 3039.0555 0.17
10 20 1000.00 4432.4011 3008.5601 0.13
Firedrake with Apptainer¶
Firedrake is a Python-based finite element package that requires a large number of dependencies.
Unfortunately, the installer provided by the developers is only intended for Ubuntu and MacOS X systems, so deploying it natively on DelftBlue in a maintainable is virtually impossible. We therefore recommend running it in an Apptainer container. Other users have made such containers available, issue the following commands on DelftBlue to see a list:
Let's pick one of them and build it locally. Note that you must set TMPDIR because /tmp cannot be used on DelftBlue.