Using Python Virtual Environments on the Devana Supercomputer¶

This document outlines how to set up and use Python virtual environments (venv) and virtualenvwrapper for managing dependencies in your projects on the Devana supercomputer.

Introduction to Virtual Environments¶

A virtual environment is an isolated Python workspace that lets you install project-specific dependencies separate from the system Python. This approach helps prevent conflicts and supports reproducibility. While similar to conda environments used in Anaconda, virtual environments are more lightweight and come built into Python. Unlike conda, they do not include binary packages or extensive dependency management but are adequate for most projects.

Benefits of Virtual Environments¶

Isolated dependencies: Keeps your project's libraries separate from system libraries.
Reproducibility: Simplifies sharing and deploying code with consistent dependencies.
Permission management: Installs packages without requiring administrator rights.

Using `venv`¶

The venv module is built into Python and can be used to create virtual environments.

Creating a Virtual Environment¶

First one has to load apropriate python interpreter:
```
module load Python/<version>
```
Create a virtual environment in desired directory:
```
python3 -m venv /path/to/the/directory
```

Activate the environment:

source /path/to/the/directory/bin/activate

Deactivate the environmet:
```
deactivate
```

Installing Packages¶

Use pip within the activated virtual environment:

pip install <package-name>

The package will be available as long as the environment will be loaded, or removed from the virtual environment.

Using Python Virtual Environments in SLURM Batch scripts¶

In order to use Python Virtual Environments in SLURM Batch script one has to load Appropriate python version and source the environment it self.

module load Python/<version>
source /path/to/the/directory/bin/activate

Troubleshooting¶

The most common error can be if loaded python version does not corespond to the python version which is used in the environment. When a virtual environment is created a symbolic link to the specific python is established. When incorrect python version is loaded with respect to the environment the desired work will might work but it may fail at some point. Therefore, it is important to be sure rith python is loaded with right environment. To ensure that one can add following line in the beginning of the activate script:

module load Python/<version>

Using `virtualenvwrapper`¶

virtualenvwrapper is a tool that provides additional utilities to simplify the management of multiple virtual environments. Think of it as an organizer for your Python virtual environments.

Setup `virtualenvwrapper`¶

One has to setup WORKON_HOME environment variable and source the setup script. As they are suggested here:

export WORKON_HOME=$HOME/envs
source $HOME/.local/bin/virtualenvwrapper.sh

The WORKON_HOME variable stores the absolute path to the directory where the virtual environment will be stored. second source sources the the bash functions needed for virtualenvrapper to work. It is suggested to put these two commands to you .bashrc script stored in you home directory.

Basic Commands¶

Here are most basic commands with virtualenvwrapper:

# Create new virtual environment with selected flavor of python
mkvirtualenv -p $(which python3) <name-of-the-environment>

# Activate environment
workon <name-of-the-environment>

# Deactivate environment
deactivate

# Remove environment
rmvirtualenv <name-of-the-environment>

Best Practices¶

Create a separate environment for each project.
After activating the environment one can easilly jump to the environment directory by:
```
cd $VIRTUAL_ENV
```
It is suggested to make some alias for this command.
In the $VIRTUAL_ENV/bin directory four scripts (hooks) are stored:
- preactivate Executed before the activation of environment
- postactivate Executed after the activation of environment
- predeactivate Executed before the deactivation of environment
- postdeactivate Executed after the deactivation of environment
One can utilize these script as one feels it is necessary for their project.
Using Hook Scripts
```
echo "module load Python/<version>" >> $VIRTUAL_ENV/bin/preactivate
echo "echo 'Environment activated!'" >> $VIRTUAL_ENV/bin/postactivate
```
- Save dependencies to requirements.txt for reproducibility:
```
pip freeze > requirements.txt
```
- Reinstall dependencies using requirements.txt:
```
pip install -r requirements.txt
```

Troubleshooting

Environment Activation Fails: Ensure the source command is used and paths are correct.
Package Installation Errors: Check that the correct Python interpreter is active.

Created by: Marek Štekláč