Skip to content

Connecting OpenAI-Compatible Clients

vLLM provides an OpenAI-compatible HTTP API. This allows different applications, scripts, and tools to use a model running on a Perun GPU node without requiring a dedicated integration for each client.

Since the vLLM server listens on the compute node, clients must access it through an SSH tunnel. The tunnel forwards a local port on the client machine to the API port on the compute node.

Connection parameters

The connection requires the following information:

Parameter Description
NODE Compute node running the vLLM server
SERVER_PORT Port used by the vLLM server
LOCAL_PORT Local port used by the SSH tunnel
API_KEY API key configured when starting vLLM
SERVED_MODEL Model name exposed by the vLLM server

The compute node and server port are stored in the metadata file created when the vLLM SLURM job starts.

The client connects to the following local API endpoint:

http://127.0.0.1:LOCAL_PORT/v1

Establishing the SSH tunnel

First, load the configuration and identify the running vLLM job.

Open an SSH tunnel 
# Load the client configuration.
source ~/llm/config.env

# Specify the SLURM job running vLLM.
JOBID=<SLURM_JOB_ID>
JOB_FILE="$HOME/llm/run/${JOBID}.job"

# Verify that the job is still running.
squeue -h -j "$JOBID" -o "%T"

# Read connection details from the job metadata.
NODE=$(sed -n 's/^NODE=//p' "$JOB_FILE")
SERVER_PORT=$(sed -n 's/^SERVER_PORT=//p' "$JOB_FILE")

# Select an available local port.
LOCAL_PORT=9999

# Open the SSH tunnel.
ssh -o ExitOnForwardFailure=yes \
    -o ServerAliveInterval=60 \
    -N -L "127.0.0.1:${LOCAL_PORT}:127.0.0.1:${SERVER_PORT}" \
    "$NODE"

The SSH command remains active while the tunnel is open. Keep this terminal running for the entire client session.

The local port can be changed if it is already in use. It does not need to match the server port.

Testing the connection

Before configuring an application, verify that the API is reachable.

Check available models 
# Load the API key.
source ~/llm/config.env

# Query the vLLM model endpoint through the SSH tunnel.
curl -sS http://127.0.0.1:9999/v1/models \
    -H "Authorization: Bearer ${API_KEY}"

A successful response contains information about the model served by vLLM. If the request fails, check that the SLURM job is running, the SSH tunnel is active, and the API key is correct.

Using a Python client

The official OpenAI Python SDK can communicate with vLLM by changing the API base URL and providing the corresponding API key.

Install the client library if it is not already available:

pip install openai

The following example can be used from a Python script or Jupyter notebook.

Python client 
import os
from openai import OpenAI

# Configure the OpenAI-compatible client.
client = OpenAI(
    base_url="http://127.0.0.1:9999/v1",
    api_key=os.environ["API_KEY"],
)

# Send a chat completion request.
response = client.chat.completions.create(
    model=os.environ["SERVED_MODEL"],
    messages=[
        {
            "role": "user",
            "content": "Summarise this log file.",
        }
    ],
)

# Print the generated response.
print(response.choices[0].message.content)

Make sure the environment variables are available to the Python process. They can be loaded from config.env before starting Python:

source ~/llm/config.env
python script.py

Using other clients

Any application supporting the OpenAI API can use the same connection parameters. Configure the client with:

  • Base URL: http://127.0.0.1:LOCAL_PORT/v1
  • API key: value of API_KEY
  • Model: value of SERVED_MODEL

The exact configuration method depends on the application. Some clients accept these values through environment variables, while others require them in a configuration file or command-line options.

The API supports standard OpenAI-compatible endpoints, including Chat Completions and Completions. Available functionality also depends on the model and its capabilities.

Closing the connection

When finished, terminate the SSH tunnel by pressing Ctrl+C in the terminal where the SSH command is running.

The vLLM server continues running until its SLURM job ends or is cancelled. Closing the SSH tunnel does not stop the server.

Important notes

  • The SLURM job must remain active while the client is using the API.
  • The SSH tunnel must remain open for the entire client session.
  • Multiple clients can use the same vLLM server, provided sufficient server capacity is available.
  • Different clients can use different local ports while connecting to the same server.
  • Do not expose the vLLM API port directly to external networks. Use the SSH tunnel and protect the API key.
  • The API key authenticates requests to protected API endpoints, but it does not secure every endpoint exposed by the vLLM server.

Further reading

Created by: Marek Štekláč