Using Python Virtual Environments¶

Python virtual environments allow users to install project-specific dependencies without affecting the system Python installation. This is particularly useful on HPC systems where users do not have administrator privileges and multiple projects may require different software versions.

The standard Python module venv provides lightweight virtual environments that isolate installed packages and dependencies.

Benefits of Virtual Environments¶

Isolated dependencies: Keeps your project's libraries separate from system libraries.
Reproducibility: Simplifies sharing and deploying code with consistent dependencies.
Permission management: Installs packages without requiring administrator rights.

Compared to Conda environments, Python virtual environments are lighter and simpler. However, Conda provides additional dependency management and binary packages.

Architecture-specific virtual environments

Perun provides both x86_64 CPU nodes and ARM64 (AArch64) GPU nodes based on NVIDIA Grace Hopper processors.

Python virtual environments are architecture-specific. A virtual environment created on the login nodes (x86_64) cannot generally be used on GPU nodes (AArch64), and vice versa.

If the environment will be used on GPU nodes, create it directly on a GPU node:

srun --partition=gpu --gres=gpu:1 --pty bash

module load Python/<version>
python3 -m venv myenv
source myenv/bin/activate

Package requirement files such as requirements.txt can be shared between architectures, but the virtual environment itself must be recreated separately on each architecture.

Using `venv`¶

The venv module is built into Python and can be used to create virtual environments.

Creating a Virtual Environment¶

First, load the appropriate Python interpreter:
```
module load Python/<version>
```
Create a virtual environment in desired directory:
```
python3 -m venv /path/to/the/directory
```

Activate the environment:

source /path/to/the/directory/bin/activate

Deactivate the environment:
```
deactivate
```

Installing Packages¶

Use pip within the activated virtual environment:

pip install <package-name>

The package is available for as long as the virtual environment is active; it is removed when the environment is deleted.

Using Python Virtual Environments in SLURM Batch scripts¶

In order to use a Python virtual environment in a SLURM batch script, one has to load the appropriate Python version and source the environment itself.

module load Python/<version>
source /path/to/the/directory/bin/activate

Troubleshooting¶

The most common problem occurs when the Python version used to create the environment differs from the Python module currently loaded.

When a virtual environment is created, it stores a symbolic link to the Python interpreter. If a different Python module is loaded later, the environment may behave unpredictably.

To avoid this issue, ensure that the same Python module is always loaded before activating the environment.

Some users add the module load command directly to the activation script:

module load Python/<version>

Using `virtualenvwrapper`¶

virtualenvwrapper is a tool that provides additional utilities to simplify the management of multiple virtual environments. Think of it as an organizer for your Python virtual environments.

Setting up `virtualenvwrapper`¶

One has to set up the WORKON_HOME environment variable and source the setup script, as shown here:

export WORKON_HOME=$HOME/envs
source $HOME/.local/bin/virtualenvwrapper.sh

The WORKON_HOME variable stores the absolute path to the directory where the virtual environment will be stored. The second source sources the bash functions needed for virtualenvwrapper to work. It is suggested to put these two commands in your .bashrc file stored in your home directory.

Basic Commands¶

Here are the most basic commands with virtualenvwrapper:

# Create new virtual environment with selected flavor of python
mkvirtualenv -p $(which python3) <name-of-the-environment>

# Activate environment
workon <name-of-the-environment>

# Deactivate environment
deactivate

# Remove environment
rmvirtualenv <name-of-the-environment>

Best Practices¶

Create a separate environment for each project.
After activating the environment, one can easily jump to the environment directory by:
```
cd $VIRTUAL_ENV
```
It is suggested to create an alias for this command.
In the $VIRTUAL_ENV/bin directory four scripts (hooks) are stored:
- preactivate Executed before the activation of environment
- postactivate Executed after the activation of environment
- predeactivate Executed before the deactivation of environment
- postdeactivate Executed after the deactivation of environment
One can utilize these script as one feels it is necessary for their project.
Using Hook Scripts
```
echo "module load Python/<version>" >> $VIRTUAL_ENV/bin/preactivate
echo "echo 'Environment activated!'" >> $VIRTUAL_ENV/bin/postactivate
```
- Save dependencies to requirements.txt for reproducibility:
```
pip freeze > requirements.txt
```
- Reinstall dependencies using requirements.txt:
```
pip install -r requirements.txt
```

Troubleshooting

Environment Activation Fails: Ensure the source command is used and paths are correct.
Package Installation Errors: Check that the correct Python interpreter is active.

Module and architecture requirements

If you use a Python module to create a virtual environment, the same Python module must be loaded before activating the environment.

On Perun, virtual environments are also architecture-specific. Environments created on x86_64 login or CPU nodes cannot generally be used on ARM64 Grace Hopper GPU nodes. Create separate virtual environments for each architecture when required.

Created by: Marek Štekláč, Marek Štekláč, Oto Kohulák