Using Python Virtual Environments¶

Python virtual environments allow users to install project-specific dependencies without affecting the system Python installation. This is particularly useful on HPC systems where users do not have administrator privileges and multiple projects may require different software versions.

The standard Python module venv provides lightweight virtual environments that isolate installed packages and dependencies.

Benefits of Virtual Environments¶

Isolated dependencies: Keeps your project's libraries separate from system libraries.
Reproducibility: Simplifies sharing and deploying code with consistent dependencies.
Permission management: Installs packages without requiring administrator rights.

Compared to Conda environments, Python virtual environments are lighter and simpler. However, Conda provides additional dependency management and binary packages.

Using `venv`¶

The venv module is built into Python and can be used to create virtual environments.

Creating a Virtual Environment¶

First one has to load apropriate python interpreter:
```
module load Python/<version>
```
Create a virtual environment in desired directory:
```
python3 -m venv /path/to/the/directory
```

Activate the environment:

source /path/to/the/directory/bin/activate

Deactivate the environmet:
```
deactivate
```

Installing Packages¶

Use pip within the activated virtual environment:

pip install <package-name>

The package will be available as long as the environment will be loaded, or removed from the virtual environment.

Using Python Virtual Environments in SLURM Batch scripts¶

In order to use Python Virtual Environments in SLURM Batch script one has to load Appropriate python version and source the environment it self.

module load Python/<version>
source /path/to/the/directory/bin/activate

Troubleshooting¶

The most common problem occurs when the Python version used to create the environment differs from the Python module currently loaded.

When a virtual environment is created, it stores a symbolic link to the Python interpreter. If a different Python module is loaded later, the environment may behave unpredictably.

To avoid this issue, ensure that the same Python module is always loaded before activating the environment.

Some users add the module load command directly to the activation script:

module load Python/<version>

Using `virtualenvwrapper`¶

virtualenvwrapper is a tool that provides additional utilities to simplify the management of multiple virtual environments. Think of it as an organizer for your Python virtual environments.

Setup `virtualenvwrapper`¶

One has to setup WORKON_HOME environment variable and source the setup script. As they are suggested here:

export WORKON_HOME=$HOME/envs
source $HOME/.local/bin/virtualenvwrapper.sh

The WORKON_HOME variable stores the absolute path to the directory where the virtual environment will be stored. second source sources the the bash functions needed for virtualenvrapper to work. It is suggested to put these two commands to you .bashrc script stored in you home directory.

Basic Commands¶

Here are most basic commands with virtualenvwrapper:

# Create new virtual environment with selected flavor of python
mkvirtualenv -p $(which python3) <name-of-the-environment>

# Activate environment
workon <name-of-the-environment>

# Deactivate environment
deactivate

# Remove environment
rmvirtualenv <name-of-the-environment>

Best Practices¶

Create a separate environment for each project.
After activating the environment one can easilly jump to the environment directory by:
```
cd $VIRTUAL_ENV
```
It is suggested to make some alias for this command.
In the $VIRTUAL_ENV/bin directory four scripts (hooks) are stored:
- preactivate Executed before the activation of environment
- postactivate Executed after the activation of environment
- predeactivate Executed before the deactivation of environment
- postdeactivate Executed after the deactivation of environment
One can utilize these script as one feels it is necessary for their project.
Using Hook Scripts
```
echo "module load Python/<version>" >> $VIRTUAL_ENV/bin/preactivate
echo "echo 'Environment activated!'" >> $VIRTUAL_ENV/bin/postactivate
```
- Save dependencies to requirements.txt for reproducibility:
```
pip freeze > requirements.txt
```
- Reinstall dependencies using requirements.txt:
```
pip install -r requirements.txt
```

Troubleshooting

Environment Activation Fails: Ensure the source command is used and paths are correct.
Package Installation Errors: Check that the correct Python interpreter is active.

Created by: Andrej Sec