Deploy vLLM on a VPS

vLLM is a high-throughput inference server for large language models. Its PagedAttention memory manager and continuous batching engine deliver far better concurrency than naive inference loops, and it exposes an OpenAI-compatible API that drops straight into existing tooling.

vLLM is best known as a GPU server, but it ships a fully supported x86 CPU backend. RamNode Cloud VPS instances are CPU-only KVM, so this guide covers the CPU path. That is a real constraint, and this guide is honest about it: you are trading tokens per second for control, privacy, and a predictable monthly bill. Read the sizing section before you provision anything.

Set Expectations First

On a modern CPU with AVX-512, expect roughly the following for a small instruct model in bfloat16:

Those numbers are order-of-magnitude, not benchmarks. Your actual result depends on the host CPU generation, core count, and memory bandwidth, which is usually the real bottleneck on virtualized hardware.

vLLM on CPU is a good fit for:

• Embedding and reranking models, where CPU is genuinely competitive
• Small classification and extraction models running as internal services
• Low-concurrency internal tools and agents where a few seconds of latency is fine
• Development and integration testing against the exact API surface you will use in production
• Batch jobs where total throughput matters more than per-request latency

It is a poor fit for interactive chat with a 7B+ model, or for anything user-facing that needs sub-second first tokens. If you want general chat on CPU, Ollama with a Q4 GGUF quant will give you better tokens per second, because llama.cpp is more aggressively optimized for quantized CPU inference. Choose vLLM when you specifically want its API surface, its batching behavior, or a dev environment that mirrors a GPU deployment elsewhere.

Server Requirements

Budget disk for model weights. An 8B model in bfloat16 is roughly 16 GB on disk, plus the Hugging Face cache keeps the original download. Provision at least three times your model size.

Deploy Ubuntu 24.04 LTS. Everything below assumes a sudo user with SSH keys.

Step 1: Verify Your CPU

vLLM's x86 CPU backend performs dramatically better with AVX-512, and best of all with AMX on very recent Xeon parts. Check what your instance actually landed on:

lscpu | grep -E "Model name|^CPU\(s\)|Thread"
lscpu | grep -o -E "avx512[a-z_]*|avx2|amx[a-z_]*" | sort -u

If you see avx512f and avx512_bf16, you are in good shape. If you only see avx2, the backend will still build and run, but expect meaningfully lower throughput and plan on smaller models. If you see neither, stop here and use a different inference stack.

RamNode instances land on different host generations depending on region and plan. Because billing is hourly, provision one, run the check above, and destroy it if the CPU is not what you need. That costs pennies.

Step 2: Prepare the System

sudo apt update && sudo apt upgrade -y
sudo apt install -y build-essential cmake ninja-build git curl \
  python3 python3-venv python3-dev libnuma-dev numactl \
  gcc-12 g++-12 libtcmalloc-minimal4

The CPU backend wants GCC 12 or newer. Set it as the default for the build:

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 120 \
  --slave /usr/bin/g++ g++ /usr/bin/g++-12

Add swap. Model loading briefly spikes memory well above steady-state, and the OOM killer will end your build otherwise:

sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo sysctl -w vm.swappiness=10

Create a service account:

sudo useradd --system --create-home --home-dir /opt/vllm --shell /bin/bash vllm
sudo mkdir -p /opt/vllm/{models,logs}
sudo chown -R vllm:vllm /opt/vllm

Step 3: Install vLLM

There is no prebuilt CPU wheel on PyPI. You have two options.

Option A: Docker (recommended)

The official CPU Dockerfile handles every build flag correctly and isolates the toolchain from your host.

sudo apt install -y docker.io docker-compose-v2
sudo usermod -aG docker $USER
newgrp docker

git clone https://github.com/vllm-project/vllm.git /opt/vllm/src
cd /opt/vllm/src
docker build -f docker/Dockerfile.cpu -t vllm-cpu:local --shm-size=4g .

The build takes 20 to 60 minutes on a small instance. Run it inside tmux or screen so an SSH drop does not kill it.

Option B: Build from source into a venv

sudo -u vllm -H bash
cd /opt/vllm
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip wheel setuptools setuptools_scm

git clone https://github.com/vllm-project/vllm.git src
cd src
pip install -r requirements/cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
VLLM_TARGET_DEVICE=cpu pip install --no-build-isolation -e .

Check out a release tag rather than main for anything you intend to keep running:

git checkout v0.11.0

Verify the install:

python -c "import vllm; print(vllm.__version__)"

Step 4: Download a Model

Pull weights ahead of time so the service does not download on every start.

sudo -u vllm -H bash
source /opt/vllm/venv/bin/activate
pip install huggingface_hub
export HF_HOME=/opt/vllm/models

huggingface-cli download Qwen/Qwen2.5-1.5B-Instruct

Good CPU-friendly starting points:

• Qwen/Qwen2.5-1.5B-Instruct for general instruction following
• meta-llama/Llama-3.2-1B-Instruct for a smaller, faster option (gated, requires accepting the license)
• BAAI/bge-small-en-v1.5 for embeddings, which is where CPU really shines
• Qwen/Qwen2.5-Coder-1.5B-Instruct for code completion backends

For gated repositories, authenticate first with huggingface-cli login.

Quantization support on the CPU backend is narrower than on GPU. Start with bfloat16 or float16 and only add quantization once you have a working baseline.

Step 5: Run the Server

Key environment variables for the CPU backend:

Test it interactively first:

export HF_HOME=/opt/vllm/models
export VLLM_CPU_KVCACHE_SPACE=4
export VLLM_CPU_OMP_THREADS_BIND=0-3
export OMP_NUM_THREADS=4
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4

vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --host 127.0.0.1 \
  --port 8000

Set VLLM_CPU_KVCACHE_SPACE to roughly one quarter of your free RAM after the model loads. Too small and concurrent requests get queued or rejected. Too large and you swap, which is catastrophic for latency.

Set --max-model-len to what you actually need. A 32k context window on CPU will consume enormous KV cache for no benefit if your prompts are 2k tokens.

Test from a second shell:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [{"role": "user", "content": "Explain PagedAttention in two sentences."}],
    "max_tokens": 128
  }'

Step 6: Run as a Service

Create /etc/systemd/system/vllm.service:

[Unit]
Description=vLLM OpenAI-compatible API server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=vllm
Group=vllm
WorkingDirectory=/opt/vllm
Environment="HF_HOME=/opt/vllm/models"
Environment="VLLM_CPU_KVCACHE_SPACE=4"
Environment="VLLM_CPU_OMP_THREADS_BIND=0-3"
Environment="OMP_NUM_THREADS=4"
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4"
EnvironmentFile=/opt/vllm/.env
ExecStart=/opt/vllm/venv/bin/vllm serve Qwen/Qwen2.5-1.5B-Instruct \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --host 127.0.0.1 \
  --port 8000 \
  --api-key ${VLLM_API_KEY} \
  --disable-log-requests
Restart=on-failure
RestartSec=15
TimeoutStartSec=900
LimitNOFILE=65535
NoNewPrivileges=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Store the API key separately:

sudo -u vllm tee /opt/vllm/.env >/dev/null <<EOF
VLLM_API_KEY=$(openssl rand -hex 32)
EOF
sudo chmod 600 /opt/vllm/.env

TimeoutStartSec=900 matters. Loading weights from disk on a small instance is slow, and the default timeout will kill the service mid-load and leave you debugging a phantom crash.

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now vllm
sudo journalctl -u vllm -f

Wait for the log line indicating the application startup is complete before sending traffic.

Docker Compose Alternative

If you built the Docker image, use /opt/vllm/compose.yaml:

services:
  vllm:
    image: vllm-cpu:local
    container_name: vllm
    restart: unless-stopped
    ports:
      - "127.0.0.1:8000:8000"
    volumes:
      - /opt/vllm/models:/root/.cache/huggingface
    environment:
      VLLM_CPU_KVCACHE_SPACE: "4"
      VLLM_CPU_OMP_THREADS_BIND: "0-3"
      OMP_NUM_THREADS: "4"
    shm_size: "4gb"
    command: >
      --model Qwen/Qwen2.5-1.5B-Instruct
      --dtype bfloat16
      --max-model-len 4096
      --host 0.0.0.0
      --port 8000

Bind the published port to 127.0.0.1 so the container does not bypass your host firewall via Docker's iptables rules. This is the most common way self-hosted inference endpoints end up exposed.

Step 7: Reverse Proxy and TLS

Never expose vLLM directly. Put Nginx in front for TLS, rate limiting, and timeouts appropriate to slow CPU generation.

sudo apt install -y nginx certbot python3-certbot-nginx

Create /etc/nginx/sites-available/vllm:

limit_req_zone $binary_remote_addr zone=llm:10m rate=10r/m;

server {
    listen 80;
    server_name llm.example.com;

    client_max_body_size 10M;

    location /v1/ {
        limit_req zone=llm burst=5 nodelay;

        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 900s;
        proxy_send_timeout 900s;
    }

    location /health {
        proxy_pass http://127.0.0.1:8000/health;
        access_log off;
    }
}

proxy_buffering off is required for token streaming to work. With buffering on, clients receive the whole response at once and streaming appears broken.

The 900 second timeouts are deliberate. A long generation on CPU can legitimately exceed the Nginx default of 60 seconds.

Enable and secure it:

sudo ln -s /etc/nginx/sites-available/vllm /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
sudo certbot --nginx -d llm.example.com

Lock down the firewall in the RamNode control panel: allow 22, 80, and 443, and deny everything else. Port 8000 should never be reachable from outside.

Step 8: Tune for Throughput

Pin threads to physical cores. On an 8 vCPU instance, if hyperthreading is exposed, binding to physical cores avoids contention:

lscpu -e=CPU,CORE,SOCKET

Set VLLM_CPU_OMP_THREADS_BIND to one thread per physical core. Oversubscribing threads consistently makes CPU inference slower, not faster.

Leave headroom. Do not bind vLLM to every core. Reserve one or two for the OS, Nginx, and the network stack.

Benchmark before and after each change:

cd /opt/vllm/src
python benchmarks/benchmark_serving.py \
  --backend openai-chat \
  --model Qwen/Qwen2.5-1.5B-Instruct \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/chat/completions \
  --dataset-name random \
  --num-prompts 20 \
  --request-rate 1

Record throughput, time to first token, and inter-token latency. Change one variable at a time.

Batch when you can. vLLM's continuous batching is where it beats simpler servers. Two concurrent requests will not take twice as long as one. If your workload is batch processing, send requests concurrently rather than sequentially.

Reduce max-model-len aggressively. KV cache scales linearly with context length. Cutting from 8192 to 2048 frees a large amount of memory for concurrency.

Monitoring

vLLM exposes Prometheus metrics at /metrics:

curl -s http://127.0.0.1:8000/metrics | grep -E "num_requests|time_to_first_token|gpu_cache_usage"

Watch vllm:num_requests_waiting. If it stays above zero, you are saturated and need a bigger KV cache, a smaller model, or a second instance.

Add a simple health check to your monitoring:

curl -sf http://127.0.0.1:8000/health || echo "vLLM down"

Troubleshooting

Build fails with an unsupported instruction error. Your CPU lacks the instruction set the build assumed. Confirm with lscpu. On AVX2-only hosts you may need to rebuild without AVX-512 optimizations, or move to a different instance.

Service killed during startup. Check sudo journalctl -k | grep -i oom. Either the model does not fit or VLLM_CPU_KVCACHE_SPACE is too large. Lower the KV cache first, then try a smaller model.

Extremely slow generation, single digit tokens per second on a small model. Check that LD_PRELOAD is set to tcmalloc and that thread binding is not oversubscribed. Confirm you are not swapping with vmstat 1 and watching the si/so columns. Any swap activity during inference destroys throughput.

"No available memory for the cache blocks" on startup. VLLM_CPU_KVCACHE_SPACE exceeds free RAM after loading weights. Reduce it, or reduce --max-model-len.

Streaming responses arrive all at once. proxy_buffering off is missing from your Nginx location block.

502 from Nginx after a long request. Increase proxy_read_timeout. CPU generation is slow enough to trip default timeouts routinely.

Model download fails with 401. The repository is gated. Run huggingface-cli login as the vllm user and accept the license on the model page.

Next Steps

Put LiteLLM in front of this instance to add virtual API keys, spend tracking, and fallback routing to a hosted provider when the local model is not good enough for a given request. That combination gives you cheap local inference for the easy 80 percent of calls, with automatic escalation for the rest.