Serving LLMs with vLLM

This documentation describes how to run a language model on a GPU node and talk to it over an OpenAI-compatible API. The model runs in a SLURM job you start and stop; the API is reachable only through your own SSH tunnel, so nothing leaves the cluster.

This page covers the model server. It is the foundation for several things:

• Qwen Code and other AI coding agents.
• Notebooks and scripts using the OpenAI Python SDK.
• Batch inference and evaluation runs.
• Any other client that speaks the OpenAI API.

To use the server from a coding agent, see Qwen Code — its launcher handles the tunnel for you. To use it from anything else, see Connecting a client.

Quickstart

Once the one-time setup is done:

cd ~/llm
sbatch submit_llm.sh          # start the server (takes a few minutes)

Release the GPU when you finish:

scancel JOB_ID

To use the server from a coding agent, see Qwen Code — its launcher handles the tunnel for you. To use it from anything else, see Connecting a client.

---

How it works

Login node (x86_64)                    GPU compute node (aarch64)
+-----------------------------+        +------------------------------+
| your client                 |        | SLURM job: run_vllm.sh       |
|   agent / notebook / script |        |   +----------------------+   |
|                             |        |   | vLLM SIF             |   |
|                             |        |   |   vllm serve         |   |
|                             |        |   +----------+-----------+   |
|   127.0.0.1:LOCAL_PORT      |        |   127.0.0.1:SERVER_PORT      |
+--------------|--------------+        +--------------|---------------+
               |                                      |
               +---------- ssh -L tunnel -------------+

• The server is private. vLLM binds 127.0.0.1 on the compute node, so it is reachable only through your own SSH tunnel. It is never on the network.
• The job owns the model's lifetime. When the SLURM job ends, for any reason including hitting its walltime, the next client request fails.
• One server, many clients. Several clients can share a running server if it has capacity. They do not each need their own GPU.
• Architecture differs across nodes. Login and CPU nodes are x86_64, GPU nodes are aarch64. A singularity built for one will not run on the other.

---

One-time setup

1. Create the directory layout

mkdir -p ~/llm/jobs ~/llm/run
chmod 700 ~/llm/run

Path	Holds
~/llm/config.env	Shared configuration, used by every script
~/llm/submit_llm.sh	SLURM submission script
~/llm/run_vllm.sh	Server launcher, runs inside the job
~/llm/jobs/	Job logs, vllm-JOBID.log
~/llm/run/	Per-job connection metadata, mode 0700

2. Generate an API key

vLLM requires a key on every request. It stops other users on the compute node from using your GPU through your port.

openssl rand -hex 24 > ~/.llm_api_key
chmod 600 ~/.llm_api_key

config.env reads the key from this file, so the key is never in the config itself, never in a SIF image, and never in a repository.

3. Install the scripts

Copy config.env, submit_llm.sh and run_vllm.sh from The scripts into ~/llm, then edit:

• config.env — Modify the LLM_PROJECT and VLLM_SIF variables.

chmod +x ~/llm/*.sh

4. Pre-download the model

Download on a login node, which has outbound network access, before the first job runs:

source ~/llm/config.env
singularity exec "$VLLM_SIF" \
    hf download "$MODEL_NAME" --cache-dir "$HF_HUB_CACHE"

Doing this inside the job wastes GPU time, and fails outright if compute nodes have no outbound access. Gated models additionally need huggingface-cli login or HF_TOKEN set. A 27B FP8 model requires roughly 30 GB of data space, and the cache accumulates across models.

---

The scripts

Three files: config.env holds settings, submit_llm.sh asks SLURM for a GPU, run_vllm.sh serves the model.

config.env — shared configuration

Every script on both this page and the agent page sources this file, so they cannot disagree about which model, which cache or which container is in use. It contains no secrets: the API key is read at the bottom from ~/.llm_api_key, which is mode 0600 and lives outside the script directory. That separation means you can copy config.env to a colleague, commit it, or paste it into a ticket without leaking anything.

~/llm/config.env

# Shared configuration for vLLM and its clients on Perun.
# Sourced by run_vllm.sh and agent.sh:   source ~/llm/config.env
#
# This file holds no secrets. The API key lives in ~/.llm_api_key (0600).

# ---- Paths -------------------------------------------------------------
export LLM_HOME="${LLM_HOME:-$HOME/llm}"
export LLM_PROJECT=""                  # LLM storage - adviced to use `/project` or `/scratch` due to quota limit on `/home`
export LLM_RUNTIME="$LLM_HOME/run"     # per-job connection metadata
export LLM_LOGS="$LLM_HOME/jobs"       # SLURM job logs

# ---- Caches ------------------------------------------------------------
# Model weights land here. Keep it off $HOME.
export AI_CACHE="$LLM_PROJECT/llm/cache"
export XDG_CACHE_HOME="$AI_CACHE"
export HF_HOME="$AI_CACHE/huggingface"
export HF_HUB_CACHE="$AI_CACHE/huggingface"
export TORCH_HOME="$AI_CACHE/torch"
export TRITON_CACHE_DIR="$AI_CACHE/triton"
export CUDA_CACHE_PATH="$AI_CACHE/cuda"
export VLLM_CONFIG_ROOT="$AI_CACHE/vllm"
export FLASHINFER_CACHE_DIR="$AI_CACHE/flashinfer"

# ---- Model - Change these for other models -----------------------------
export MODEL_NAME="Qwen/Qwen3.8-27B-FP8"   # what vLLM loads - see `https://huggingface.co` for various models
export SERVED_MODEL="qwen3.8"              # what the API advertises
export MODEL_LENGTH="262144"               # Context window length
export GPU_MEMORY_UTILIZATION="0.92"       # Set hard limit to GPU memory to avoid OOM
export TENSOR_PARALLEL_SIZE="1"            # Number of used GPUs - must be same as number of GPUs requested in slurm

# Model-specific parsers - applies specifically to Qwen Code
export REASONING_PARSER="qwen3"
export TOOL_PARSER="qwen3_coder"

# ---- Server container - Change here for other images/clusters ----------
export VLLM_SIF="/apps/images/vllm/aarch64/vllm_26.04-py3-xxhash.sif"
export QWEN_SIF_X86="/apps/images/vllm/x86_64/Qwen_code_x86_64_Nov2026.sif"

# ---- API key -----------------------------------------------------------
# Create once:
#   openssl rand -hex 24 > ~/.llm_api_key && chmod 600 ~/.llm_api_key
if [[ -r "$HOME/.llm_api_key" ]]; then
    KEY=$(< "$HOME/.llm_api_key")
    export KEY
fi

# The server port is chosen by run_vllm.sh and published in $LLM_RUNTIME.
# The local tunnel port is chosen by the client. Neither is configured here.

Devana cluster

On Devana cluster singularity containers can be found at /

submit_llm.sh — the resource request

This script does nothing but ask SLURM for a GPU and hand control to run_vllm.sh. Keeping it this thin means you can change how the server is launched without resubmitting through a different script, and you can run run_vllm.sh under srun for debugging.

~/llm/submit_llm.sh

#!/bin/bash
# Submit with:  sbatch submit_llm.sh
#
# #SBATCH lines are read by SLURM before any shell expansion, so $HOME and
# $USER do NOT work here. Edit the absolute paths below to your own login.

#SBATCH --account=                    # Slurm account
#SBATCH --job-name=vllm
#SBATCH --partition=gpu_short
#SBATCH --gres=gpu:1                  # Request one GPU
#SBATCH --ntasks=1                    
#SBATCH --cpus-per-task=72            # Corresponds to one GPU
#SBATCH --time=04:00:00
#SBATCH --output=jobs/%x-%j.log
#SBATCH --error=jobs/%x-%j.log

bash run_vllm.sh

run_vllm.sh — the model server

This runs inside the allocation, on the GPU node. In order, it:

1. Sources the configuration and refuses to continue without an API key or a SLURM_JOB_ID — the latter catches the common mistake of running it directly on a login node, where it would find no GPU.
2. Registers a cleanup trap for the metadata file, so a cancelled or crashed job does not leave a file claiming a server is running.
3. Asks the kernel for a free port preventing collisions between jobs.
4. Publishes the connection details to ~/llm/run/JOBID.job under umask 077, so only you can read where your server is.
5. Builds the argument list, adding the reasoning and tool-call parsers only when they are configured.

~/llm/run_vllm.sh

#!/bin/bash
# Runs vLLM inside a detached tmux session within the SLURM allocation.
# SLURM job remains active while the tmux session is running.
set -euo pipefail

# Load configuration
source "${LLM_HOME:-$HOME/llm02}/config.env"

# Validate required variables
: "${KEY:?No API key. Create ~/.llm_api_key -- see config.env.}"
: "${SLURM_JOB_ID:?Run this through sbatch, not directly.}"

# Prepare runtime directories
mkdir -p "$LLM_RUNTIME" "$LLM_LOGS"
chmod 700 "$LLM_RUNTIME"

# Use a job-specific session name to allow multiple vLLM jobs per user
SESSION="vllm-${SLURM_JOB_ID}"

# Metadata file
META="$LLM_RUNTIME/$SLURM_JOB_ID.job"

# Remove metadata when the SLURM job exits
trap 'rm -f "$META"' EXIT

# Let the kernel pick a free port on this compute node
SERVER_PORT=$(python3 -c '
import socket
with socket.socket() as s:
    s.bind(("127.0.0.1", 0))
    print(s.getsockname()[1])
')

# Export the selected port for other scripts
export SERVER_PORT

# Write job metadata
umask 077
cat > "$META" <<EOF
JOB_ID=$SLURM_JOB_ID
NODE=$(hostname -s)
SERVER_PORT=$SERVER_PORT
SERVED_MODEL=$SERVED_MODEL
MODEL_NAME=$MODEL_NAME
STARTED=$(date -Is)
EOF

echo "Starting vLLM: $(hostname -s):$SERVER_PORT"
echo "Model: $MODEL_NAME"
echo "Session: $SESSION"

# Build vLLM arguments
args=(
    --served-model-name "$SERVED_MODEL"
    --host 127.0.0.1
    --port "$SERVER_PORT"
    --tensor-parallel-size "$TENSOR_PARALLEL_SIZE"
    --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
    --max-model-len "$MODEL_LENGTH"
    --enable-prefix-caching
    --prefix-caching-hash-algo xxhash
)

# Optional reasoning parser
if [[ -n "${REASONING_PARSER:-}" ]]; then
    args+=(--reasoning-parser "$REASONING_PARSER")
fi

# Optional tool calling support
if [[ -n "${TOOL_PARSER:-}" ]]; then
    args+=(--enable-auto-tool-choice)
    args+=(--tool-call-parser "$TOOL_PARSER")
fi

# Optional chat template
if [[ -n "${CHAT_TEMPLATE:-}" ]]; then
    args+=(--chat-template "$CHAT_TEMPLATE")
fi

# Export API key through the tmux session environment
# This avoids exposing the key in the process command line
export VLLM_API_KEY="$KEY"

# Prepare command with proper shell escaping
printf -v MODEL_Q '%q' "$MODEL_NAME"
printf -v IMAGE_Q '%q' "$VLLM_SIF"
printf -v CACHE_Q '%q' "$AI_CACHE"

CMD="
module load singularity
unset LD_PRELOAD


singularity exec --nv \
    --env PYTHONNOUSERSITE=1 \
    --bind ${CACHE_Q}:${CACHE_Q} \
    ${IMAGE_Q} \
    vllm serve ${MODEL_Q}"

# Append arguments with shell-safe quoting
for arg in "${args[@]}"; do
    printf -v ARG_Q '%q' "$arg"
    CMD+=" $ARG_Q"
done

# Replace any existing session with the same job ID
tmux kill-session -t "$SESSION" 2>/dev/null || true

# Start detached tmux session with the API key in its environment
tmux new-session -d -s "$SESSION" -e "VLLM_API_KEY=$VLLM_API_KEY"

# Execute vLLM inside tmux
tmux send-keys -t "$SESSION" "$CMD" C-m

echo "vLLM running in tmux session: $SESSION"
echo "Node: $(hostname -s)"
echo "Port: $SERVER_PORT"
echo "Metadata: $META"

# Keep SLURM allocation alive while tmux session exists
while tmux has-session -t "$SESSION" 2>/dev/null; do
    sleep 30
done

echo "vLLM tmux session terminated."

---

Running the server

Starting

cd ~/llm
sbatch submit_llm.sh
squeue -u "$USER"

Loading takes a few minutes, longer on a cold cache. Follow it with:

tail -f ~/llm/jobs/vllm-JOBID.log

The server is ready when the log shows the API routes and Application startup complete.

Verifying it works

Before debugging any client, confirm the model answers. This cleanly separates a server problem from a client problem.

Health check from a login node

source ~/llm/config.env

# Read the node and port the job published.
NODE=$(sed -n 's/^NODE=//p'        ~/llm/run/JOBID.job)
PORT=$(sed -n 's/^SERVER_PORT=//p' ~/llm/run/JOBID.job)

ssh -N -L 9999:127.0.0.1:$PORT $NODE &
TUNNEL=$!

# 1. Is the model loaded and advertised?
curl -s -H "Authorization: Bearer $KEY" http://127.0.0.1:9999/v1/models

# 2. Does it generate?
curl -s http://127.0.0.1:9999/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"$SERVED_MODEL\",
       \"messages\":[{\"role\":\"user\",\"content\":\"Say hello.\"}]}"

kill $TUNNEL

If /v1/models lists your model and the completion returns text, the server is healthy.

Stopping

scancel JOB_ID

---

Ports

Both ports are allocated automatically. The mechanism is worth understanding because it is where most connection problems live.

Server port run_vllm.sh binds port 0 on the compute node, lets the kernel pick a free one, and publishes the result in the job's metadata file. Collisions between jobs are impossible by construction.

Local port. Each client does the same on the login node, so several sessions can run side by side.

The metadata file

One file per job, written by the server and read by every client:

~/llm/run/123456.job

JOB_ID=123456
NODE=gn003
SERVER_PORT=18013
SERVED_MODEL=qwen3.8
MODEL_NAME=Qwen/Qwen3.8-27B-FP8
STARTED=2026-10-28T10:14:02+01:00

It is created once the port is known and removed when the job exits. Clients should parse it, not source it, and should confirm the job is still running before trusting it — a file in a writable directory ought not be able to execute code, and a job can die without removing its file.

---

Connecting a client

Any OpenAI-compatible client works. The contract is the same in every case:

1. Read NODE and SERVER_PORT from the job's metadata file.
2. Check the job is still running (squeue -h -j JOBID).
3. Pick a free local port.
4. Open ssh -N -L 127.0.0.1:LOCAL:127.0.0.1:SERVER_PORT NODE.
5. Point the client at http://127.0.0.1:LOCAL/v1 with $KEY as the API key and $SERVED_MODEL as the model name.
6. Close the tunnel when the client exits.

Qwen Code ships a launcher that does all six steps. For everything else, do it by hand:

From a Python script or notebook

# Terminal 1 — open the tunnel.
source ~/llm/config.env
NODE=$(sed -n 's/^NODE=//p'        ~/llm/run/JOBID.job)
PORT=$(sed -n 's/^SERVER_PORT=//p' ~/llm/run/JOBID.job)
ssh -N -L 9999:127.0.0.1:$PORT $NODE

# Terminal 2 — talk to it.
import os
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:9999/v1",
    api_key=os.environ["KEY"],
)

reply = client.chat.completions.create(
    model=os.environ["SERVED_MODEL"],
    messages=[{"role": "user", "content": "Summarise this log file."}],
)
print(reply.choices[0].message.content)

---

Choosing a model

MODEL_NAME is what vLLM loads; SERVED_MODEL is the name the API advertises and the client must request. They need not to match.

export MODEL_NAME="organization/model-repository"
export SERVED_MODEL="my-model"

Before committing to a model, check that:

• the installed vLLM version supports its architecture,
• the weights are accessible to your account (some need a licence acceptance),
• it fits in GPU memory alongside the KV cache at your context length,
• it supports tool calling, if agents will use it,
• its chat template is present, or supply one.

Memory is dominated by weights plus KV cache, and the cache grows with MODEL_LENGTH. A 262144-token context is expensive; if a model will not fit, reduce MODEL_LENGTH before reaching for GPU_MEMORY_UTILIZATION.

Tool calling decides whether agents work

Without it a model can discuss code but cannot read or change files. It needs support in the model, in the chat template, and a matching --tool-call-parser in vLLM. If you are serving a model for agent use, verify this.

---

Customising the vLLM image

You rarely need to. Changing models needs only config.env and a restart.

Build a custom server image for a different vLLM version, extra Python packages, or model-specific dependencies. Start from a compatible CUDA base image, install a vLLM version suitable for the GPU architecture, and build on aarch64 for Perun's GPU nodes or x86_64 for Devana's GPU nodes. Keep the weights outside the image — they belong in the Hugging Face cache, bound in at runtime, thus keeping the image minimal.

---

Troubleshooting

The job starts and immediately exits

Check the log for the failure:

tail -50 ~/llm/jobs/vllm-JOBID.log

The usual causes are a model the installed vLLM cannot load, a bad MODEL_NAME, or missing weights with no network access to fetch them.

No metadata file appears

The job has not got far enough to publish its port, or it died during startup. A job in PD is queued, not broken:

squeue -u "$USER"

The model does not fit in GPU memory

The log names the allocation that failed. In order of preference: lower MODEL_LENGTH (the KV cache usually dominates), then GPU_MEMORY_UTILIZATION, then pick a smaller or more quantised model. Raising GPU_MEMORY_UTILIZATION above about 0.95 tends to trade a clean startup failure for a crash under load.

Loading takes longer than expected

A cold cache on a large model can take many minutes. Pre-download the weights on a login node, then start the job. If the log stalls partway with no error, the cause is usually memory — see above.

`401 Unauthorized`

The client is sending the wrong key, or none. Both sides read ~/.llm_api_key through config.env; confirm the file exists, is readable, and that you sourced config.env in the shell you are testing from.

`Address already in use`

With kernel-allocated ports this should not happen. If it does, something bound the port in the gap between it being released and SSH or vLLM claiming it — retry. If it recurs, check for a stale tunnel:

pgrep -af "ssh.*-L 127.0.0.1"

The client stops working mid-session

Usually the job hit its walltime:

sacct -j JOBID --format=JobID,State,Elapsed,Timelimit

Submit a new job and reconnect.

---

Reference

Configuration variables

Variable	Purpose
MODEL_NAME	Hugging Face repository vLLM loads
SERVED_MODEL	Name the API advertises; clients must request it
MODEL_LENGTH	Maximum context in tokens; drives KV cache size
GPU_MEMORY_UTILIZATION	Fraction of GPU memory vLLM may use
TENSOR_PARALLEL_SIZE	GPUs per server; must match --gres
REASONING_PARSER	Model-specific; leave unset unless required
TOOL_PARSER	Required for agent file editing; model-specific
AI_CACHE	Root of all caches, including model weights
VLLM_SIF	Server image, aarch64
QWEN_SIF_X86	Qwen Code client image, x86_64
KEY	API key, read from ~/.llm_api_key

Commands

Task	Command
Start a server	sbatch submit_llm.sh
List your jobs	squeue -u "$USER"
Follow a job log	tail -f ~/llm/jobs/vllm-JOBID.log
See where a server landed	cat ~/llm/run/JOBID.job
Stop a server	scancel JOBID
Inspect a finished job	sacct -j JOBID --format=JobID,State,Elapsed

Links

• vLLM · documentation
• Hugging Face models
• Apptainer documentation
• Qwen Code on Perun

Created by: Marek Štekláč