Compiling Software on the Cluster¶
There are three main ways to compile software on the Perun cluster. One has to keep in mind is, there are two architectures. Login nodes and CPU nodes have x86, whereas GPU have aarch64. So unless one uses cross-compiling, the builds has to be done on target machines. Compiling headers are, however, missing on compute nodes, so compiling on those can be difficult.
1. Compile on a login node¶
For most users, this is the easiest option.
Login nodes already provide the compilers, development libraries, and system headers needed to build software.
cmake -S . -B build
cmake --build build -j 4
This is a good choice if you compile software only occasionally. One has tou use cross-compiling for building executbales for GPU paritition.
Resource limits
Login nodes are shared between many users and are subject to resource limits.
2. Use the testing partition¶
Another option is to request resources from the testing partition.
This allows compilation to run through the scheduler while still using nodes that provide the normal development environment and system headers.
This is useful for builds that require more resources than should be used directly on a login node.
The disadvantage is that only a limited number of nodes are available in this partition. If other users are already using them, your job may have to wait. And again, unless the cross-compiling is used, you cannot use builds on GPU partition.
3. Compile on a compute node¶
For larger builds or for just-in-time (JIT) compilations, you can request a regular compute node.
This provides significantly more CPU and memory resources and avoids putting compilation load on the login nodes.
However, regular compute nodes contain only the runtime environment. Development packages and system headers are intentionally not installed there.
To compile software on a compute node, load the build-tools module:
module load build-tools
The module provides an Singularity-based development environment containing the required compilers, headers, and development libraries. Keep in mind these can be use on both CPU compute nodes as well as GPU compute nodes.
Interactive build environment¶
Start an interactive development shell with:
build-shell
For example:
module load build-tools
build-shell
module load GCC/15.2.0
cmake -S . -B build
cmake --build build -j 320 # or 288 on GPU nodes
The current directory as well as /project and /apps are available inside the container, so build files are written directly to the normal cluster filesystem.
Leave the build environment with:
exit
Load you toolchain
The container image does not contain the full toolchain. One still has to load the desired toolchain from the Environment Modules (i.e.: module load ...)
Compiling on parallel file systems
Compiling on parallel file systems can be slow and may cause problems with some build systems, especially when using many parallel jobs and small object files. On compute nodes, you can use local storage for temporary build files /work. Just keep in mind after the build is done, you have to copy the final binaries to a permanent location, since /work is purged after the job ends.
mkdir /work/$SLURM_JOB_ID/build
cmake -S . -B /work/$SLURM_JOBID/build
cmake --build /work/$SLURM_JOBID/build -j 72 # on one GPU card
cp -rf /work/$SLURM_JOB_ID/build/* /project/your-project/
Running a single command¶
Instead of starting an interactive shell, you can execute individual commands inside the build environment using build-env:
build-env cmake -S . -B build
build-env cmake --build build -j 320
Which option should I use?¶
| Option | Best for | Development headers | Resources |
|---|---|---|---|
| Login node | Small and occasional builds | Yes | Limited to 4 CPU cores |
| Testing partition | Larger builds using the standard environment | Yes | Limited availability |
Compute node + build-tools |
Large or parallel builds | Default no, but provided by build-tools |
Resources requested from the scheduler |
For occasional compilation, using a login node is normally the simplest option.
For very large builds (hours) or JIT, use a compute node together with build-tools.