Skip to content

Pixi and (Mini)Conda

Install your own packages locally

Pixi and Conda are popular package managers that allow you to install software in your /home or /scratch directory, including all required dependencies. If you have no preference, we recommend pixi because it is more modern.

Warning

Please be aware: package managers such as conda are known to install a lot of tiny files locally. This is important for several reasons:

  1. These local installations might occupy a lot of space and often use your /home directory as their default destination. You might want to redirect them from /home to /scratch (see below for more info).

  2. These local installation might rely on the /tmp folder as an intermediate storage for unpacking/compiling. Please be aware that the collectively used /tmp might get overfilled! More info here.

  3. /home and /scratch rely on the parallel file system BeeGFS. While this file system provides high speed for truly parallel jobs (many processes reading/writing from/to one big file), it might struggle with processes generating a lot of tiny files. As such, installing packages via conda might take noticeably longer than you would expect. This is normal, and only manifests itself once, during the installation. Once installed, accessing these packages should be very fast.

Do you need to install a package that is not available as a module? You can for example use conda to install it locally in your own /home or /scratch directory.

As the storage space needed for these package can increase rapidly you might run out of space on your home directory. Typically the following directories are used to store your packages:

installer location purpose
conda $HOME/.conda installed and cached packages
pixi where you type ``pixi init Installed and cached packages

If you install a large amount of software and run out of storage, pixi is the easiest way to install in /scratch instead. Beware, though, that /scratch is cleaned in regular intervals.

Installing your chosen package manager

You can use the following scripts to install miniconda or pixi from the command-line on a login-node:

setup-miniconda3.sh
#!/bin/bash

# This script downloads the latest miniconda3 installer,
# installs and sets up miniconda in your home directory.
# Usage: ./setup-miniconda3.sh (on the login node of DelftBlue)

MINICONDA_URL="https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh"

#define where to install
MINICONDA_ROOT="$HOME/miniconda3"

echo "--- Installing Miniconda to $MINICONDA_ROOT ---"

# get installer script
wget $MINICONDA_URL -O miniconda.sh
chmod u+x miniconda.sh

# download and install
./miniconda.sh -b -p $MINICONDA_ROOT
$MINICONDA_ROOT/bin/conda shell.bash hook
$MINICONDA_ROOT/bin/conda init bash

# Cleanup
rm miniconda.sh

echo "--- Done. To update your current session, type 'source ~/.bashrc' ---"
echo "--- The next time you log in, conda will be automatically enabled ---"
setup-pixi.sh
#!/bin/bash

curl -fsSL https://pixi.sh/install.sh | bash

PIXI_BIN="$HOME/.pixi/bin"

if [[ ":$PATH:" != *":$PIXI_BIN:"* ]]; then
    echo 'export PATH="$HOME/.pixi/bin:$PATH"' >> ~/.bashrc
fi

echo "--- Done. To update your current session, type 'source ~/.bashrc' ---"
echo "--- The next time you log in, pixi will be automatically enabled ---"
  • Copy the appropriate script into a text file, say, setup.sh
  • make it executable using chmod u+x setup.sh
  • and run it using ./setup.sh

Afterwards, update your shell by typing source/.bashrc. Both scripts are designed to add the respective tool to your path by default in subsequent DelftBlue sessions, if you do not want this, edit them before execution or remove the lines they create in/.bashrc`` manually.

There is plenty of up-to-date information about these tools available online, so we will not duplicate that information here. If you want to learn more, here is one link per tool: