Compiling Software on the Cluster¶
There are three main ways to compile software on the Devana cluster.
1. Compile on a login node¶
For most users, this is the easiest option.
Login nodes already provide the compilers, development libraries, and system headers needed to build software.
cmake -S . -B build
cmake --build build -j 4
This is the choice if you compile software only occasionally, and most of the users do not have to go an further.
Resource limits
Login nodes are shared between many users and are subject to resource limits. Compilation is currently limited to 4 CPU cores.
2. Use the testing partition¶
Another option is to request resources from the testing partition.
This allows compilation to run through the scheduler while still using nodes that provide the normal development environment and system headers, since the run on login nodes. In case of devana testing partitoin it is 16.
The disadvantage is that only a limited number of nodes are available in this partition (only 2). If other users are already using them, your job may have to wait. Another problem can be compiling on parallel file systems, which can be slow and may cause problems with some build systems, especially when using many parallel jobs and small object files. Login nodes do not have local scratch space, so compilation on the login nodes can be slow for large builds.
3. Compile on a compute node¶
For larger builds or for just-in-time (JIT) compilations, you can request a regular compute node.
This provides significantly more CPU and memory resources and avoids putting compilation load on the login nodes.
However, regular compute nodes contain only the runtime environment. Development packages and system headers are intentionally not installed there.
To compile software on a compute node, load the build-tools module:
module load build-tools
The module provides an Singularity-based development environment containing the required system headers.
Interactive build environment¶
Start an interactive development shell with:
build-shell
For example:
module load build-tools
build-shell
module load GCC/15.2.0
cmake -S . -B build
cmake --build build -j 64
The current directory as well as /project and /storage-apps are available inside the container, so build files are written directly to the normal cluster filesystem.
Leave the build environment with:
exit
Running a single command¶
Instead of starting an interactive shell, you can execute individual commands inside the build environment using build-env:
build-env cmake -S . -B build
build-env cmake --build build -j 16
Load you toolchain
The container image does not contain the full toolchain. One still has to load the desired toolchain from the Environment Modules (i.e.: module load ...)
Compiling on parallel file systems
Compiling on parallel file systems can be slow and may cause problems with some build systems, especially when using many parallel jobs and small object files. On compute nodes, you can use local storage for temporary build files /work. Just keep in mind after the build is done, you have to copy the final binaries to a permanent location, since /work is purged after the job ends.
mkdir /work/$SLURM_JOB_ID/build
cmake -S . -B /work/$SLURM_JOBID/build
cmake --build /work/$SLURM_JOBID/build -j 64
cp -rf /work/$SLURM_JOB_ID/build/* /project/your-project/
Which option should I use?¶
| Option | Best for | Development headers | Resources |
|---|---|---|---|
| Login node | Small and occasional builds | Yes | Limited to 4 CPU cores |
| Testing partition | Larger builds using the standard environment | Yes | Limited availability |
Compute node + build-tools |
Large or parallel builds | Default no, but provided by build-tools |
Resources requested from the scheduler |
For occasional compilation, using a login node is normally the simplest option.
For very large builds (hours) or JIT, use a compute node together with build-tools.