Deploy the LiteLLM Proxy on a VPS

LiteLLM Proxy is a self-hosted gateway that puts a single OpenAI-compatible endpoint in front of every model provider you use. Point your applications at one URL, issue virtual API keys per team or project, set spend budgets, configure automatic fallbacks when a provider errors, and get unified logging across OpenAI, Anthropic, Bedrock, Azure, Vertex, and any local model server you run.

It is genuinely lightweight. A 2 vCPU RamNode instance handles a substantial amount of proxied traffic, because LiteLLM is doing routing and bookkeeping, not inference.

This guide deploys LiteLLM with Docker Compose, a PostgreSQL backend for key and spend persistence, the admin UI behind TLS, and a hardened configuration suitable for a team.

What You Will Build

LiteLLM Proxy running under Docker Compose with automatic restart
PostgreSQL 16 for virtual keys, teams, budgets, and request logs
Redis for cross-worker rate limiting and response caching
Virtual API keys with per-key budgets and model allowlists
Fallback chains so a provider outage does not take down your applications
The admin UI at /ui served over HTTPS with a strong master key
Automated backups of the key and spend database

Server Requirements

Scale	Recommended Plan
Personal or small team, under 50k requests/day	2 vCPU, 2 GB RAM, 40 GB NVMe
Team with logging and caching enabled	2 vCPU, 4 GB RAM, 80 GB NVMe
High volume, many concurrent streams	4 vCPU, 8 GB RAM, 160 GB NVMe

Request logging is what drives storage growth. If you enable full prompt and response logging, size disk generously and set a retention policy from day one.

Deploy Ubuntu 24.04 LTS with a sudo user and SSH keys configured.

Step 1: Install Docker

shell

sudo apt update && sudo apt upgrade -y
sudo apt install -y ca-certificates curl gnupg git

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
newgrp docker

Step 2: Create the Stack

shell

sudo mkdir -p /opt/litellm/{config,postgres,logs}
sudo chown -R $USER:$USER /opt/litellm
cd /opt/litellm

Generate secrets and write the environment file:

shell

cat > /opt/litellm/.env <<EOF
LITELLM_MASTER_KEY=sk-$(openssl rand -hex 32)
LITELLM_SALT_KEY=sk-$(openssl rand -hex 32)
POSTGRES_PASSWORD=$(openssl rand -hex 24)
UI_USERNAME=admin
UI_PASSWORD=$(openssl rand -hex 16)

# Provider credentials
OPENAI_API_KEY=sk-your-openai-key
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
EOF

chmod 600 /opt/litellm/.env
cat /opt/litellm/.env | grep -E "MASTER_KEY|UI_PASSWORD"

Record the master key and UI password now. The salt key encrypts stored provider credentials in the database. Never change LITELLM_SALT_KEY after the first run. Doing so makes every stored credential unreadable and there is no recovery path short of re-entering them all.

Create /opt/litellm/compose.yaml:

shell

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    container_name: litellm
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_started
    ports:
      - "127.0.0.1:4000:4000"
    volumes:
      - ./config/config.yaml:/app/config.yaml:ro
    command: ["--config", "/app/config.yaml", "--port", "4000", "--num_workers", "2"]
    environment:
      DATABASE_URL: "postgresql://litellm:${POSTGRES_PASSWORD}@postgres:5432/litellm"
      STORE_MODEL_IN_DB: "True"
      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
      LITELLM_SALT_KEY: ${LITELLM_SALT_KEY}
      UI_USERNAME: ${UI_USERNAME}
      UI_PASSWORD: ${UI_PASSWORD}
      REDIS_HOST: redis
      REDIS_PORT: "6379"
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
    healthcheck:
      test: ["CMD-SHELL", "curl -sf http://localhost:4000/health/liveliness || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s

  postgres:
    image: postgres:16-alpine
    container_name: litellm-postgres
    restart: unless-stopped
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - ./postgres:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm -d litellm"]
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    container_name: litellm-redis
    restart: unless-stopped
    command: ["redis-server", "--save", "60", "1", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru"]
    volumes:
      - ./redis:/data

Binding to 127.0.0.1:4000 keeps the proxy off the public internet. Docker publishes ports by writing iptables rules that bypass most host firewalls, so the bind address is doing real security work here.

Use main-stable rather than main-latest. LiteLLM ships frequently and main-latest will change under you.

Step 3: Write the Model Configuration

Create /opt/litellm/config/config.yaml:

shell

model_list:
  # Hosted providers
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
      rpm: 500

  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
      rpm: 1000

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 300

  # Local model served by vLLM or Ollama on this network
  - model_name: local-small
    litellm_params:
      model: openai/Qwen/Qwen2.5-1.5B-Instruct
      api_base: http://172.17.0.1:8000/v1
      api_key: os.environ/VLLM_API_KEY

router_settings:
  routing_strategy: usage-based-routing-v2
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT
  num_retries: 2
  timeout: 120
  allowed_fails: 3
  cooldown_time: 60
  fallbacks:
    - gpt-4o: ["claude-sonnet"]
    - claude-sonnet: ["gpt-4o"]
    - local-small: ["gpt-4o-mini"]

litellm_settings:
  drop_params: true
  set_verbose: false
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    ttl: 3600
    supported_call_types: ["acompletion", "aembedding"]
  success_callback: ["postgres"]
  failure_callback: ["postgres"]
  max_budget: 500
  budget_duration: 30d

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  store_model_in_db: true
  disable_spend_logs: false
  proxy_batch_write_at: 60
  alerting: ["slack"]
  alerting_threshold: 300

Several settings deserve explanation.

drop_params: true silently discards parameters a given provider does not support, so a request written for OpenAI does not 400 when it falls back to Anthropic. This is what makes fallbacks actually usable.

fallbacks fire when a model errors or hits its rate limit. The local-small: ["gpt-4o-mini"] chain is the interesting one: serve cheap requests from your own hardware and escalate to a hosted provider only when the local server is down or overloaded.

cooldown_time pulls a failing deployment out of rotation for 60 seconds after allowed_fails errors, rather than hammering a provider that is already having a bad day.

proxy_batch_write_at: 60 buffers spend log writes. Without it, every request generates a database write and PostgreSQL becomes your bottleneck.

172.17.0.1 is the default Docker bridge gateway, which is how the container reaches a service listening on the host. Confirm yours with ip addr show docker0.

Step 4: Start the Stack

shell

cd /opt/litellm
docker compose up -d
docker compose logs -f litellm

LiteLLM runs database migrations on first start. Wait for the startup to complete before testing.

Verify health:

shell

curl -s http://127.0.0.1:4000/health/liveliness
curl -s http://127.0.0.1:4000/health/readiness | python3 -m json.tool

Send a test completion with the master key:

shell

source /opt/litellm/.env
curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Reply with OK."}]
  }'

Step 5: Reverse Proxy and TLS

shell

sudo apt install -y nginx certbot python3-certbot-nginx

Create /etc/nginx/sites-available/litellm:

shell

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    server_name llm-gateway.example.com;

    client_max_body_size 50M;

    location / {
        proxy_pass http://127.0.0.1:4000;
        proxy_http_version 1.1;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
    }

    location /health/liveliness {
        proxy_pass http://127.0.0.1:4000/health/liveliness;
        access_log off;
    }
}

proxy_buffering off is mandatory. With buffering enabled, streamed responses are held until complete and every streaming client in your stack appears to hang.

client_max_body_size 50M accommodates vision requests with base64 images and audio uploads.

Enable and issue a certificate:

shell

sudo ln -s /etc/nginx/sites-available/litellm /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
sudo certbot --nginx -d llm-gateway.example.com

In the RamNode control panel, allow only 22, 80, and 443. Port 4000 must never be reachable externally: anyone who finds it can spend your provider budget.

Step 6: Issue Virtual Keys

The master key is for administration only. Never hand it to an application.

Create a key scoped to a project with a budget and a model allowlist:

shell

source /opt/litellm/.env
curl -s http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "key_alias": "internal-support-bot",
    "models": ["gpt-4o-mini", "local-small"],
    "max_budget": 25,
    "budget_duration": "30d",
    "rpm_limit": 60,
    "tpm_limit": 100000,
    "metadata": {"team": "support", "owner": "vanessa"}
  }'

The response contains the generated key. Distribute that to the application, which then uses it exactly as an OpenAI key:

shell

from openai import OpenAI

client = OpenAI(
    api_key="sk-generated-virtual-key",
    base_url="https://llm-gateway.example.com/v1",
)

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Summarize this ticket."}],
)

Inspect spend on a key:

shell

curl -s "http://127.0.0.1:4000/key/info?key=sk-generated-virtual-key" \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" | python3 -m json.tool

Revoke a key:

shell

curl -s http://127.0.0.1:4000/key/delete \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"keys": ["sk-generated-virtual-key"]}'

Issue one key per application, not per person. When a key leaks, you revoke one service instead of auditing everything.

Step 7: Teams and Budgets

Create a team so several keys share a pooled budget:

shell

curl -s http://127.0.0.1:4000/team/new \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "team_alias": "engineering",
    "max_budget": 200,
    "budget_duration": "30d",
    "models": ["gpt-4o", "gpt-4o-mini", "claude-sonnet"],
    "tpm_limit": 500000
  }'

Then generate keys against the returned team_id by adding "team_id": "..." to the key generation payload. Team budgets are enforced in aggregate, so one runaway script cannot consume the whole organization's allowance.

Step 8: The Admin UI

The UI is served at /ui on the same domain. Log in with UI_USERNAME and UI_PASSWORD from your environment file.

From there you can view spend by key, team, and model, create and revoke keys without curl, add models without editing config.yaml (because store_model_in_db is on), and inspect recent request logs.

Restrict access further if the gateway is internet-facing. Add a source IP allowlist to the /ui location block:

shell

location /ui {
    allow 203.0.113.0/24;
    deny all;

    proxy_pass http://127.0.0.1:4000;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
}

Place this block before the catch-all location / block.

Step 9: Alerting and Backups

Slack Alerts

Add a webhook to your environment file and restart:

shell

echo 'SLACK_WEBHOOK_URL=https://hooks.slack.com/services/XXX/YYY/ZZZ' >> /opt/litellm/.env
docker compose up -d

With alerting: ["slack"] in config.yaml, LiteLLM notifies on budget thresholds, provider outages, and requests slower than alerting_threshold seconds.

Database Backups

The PostgreSQL database holds your virtual keys and spend history. Losing it means every application key stops working.

Create /opt/litellm/backup.sh:

shell

#!/usr/bin/env bash
set -euo pipefail

BACKUP_DIR=/var/backups/litellm
mkdir -p "$BACKUP_DIR"

docker exec litellm-postgres pg_dump -U litellm -Fc litellm \
  > "$BACKUP_DIR/litellm-$(date +%F).dump"

find "$BACKUP_DIR" -name "litellm-*.dump" -mtime +14 -delete

Schedule it:

shell

sudo chmod +x /opt/litellm/backup.sh
sudo tee /etc/cron.d/litellm-backup >/dev/null <<'EOF'
0 3 * * * root /opt/litellm/backup.sh >> /var/log/litellm-backup.log 2>&1
EOF

Back up /opt/litellm/.env separately and store it somewhere safe. Without LITELLM_SALT_KEY, a restored database cannot decrypt stored provider credentials.

Log Retention

Spend logs grow steadily. Prune old rows monthly:

shell

docker exec litellm-postgres psql -U litellm -d litellm -c \
  "DELETE FROM \"LiteLLM_SpendLogs\" WHERE \"startTime\" < NOW() - INTERVAL '90 days';"

Check current size first:

shell

docker exec litellm-postgres psql -U litellm -d litellm -c \
  "SELECT pg_size_pretty(pg_database_size('litellm'));"

Step 10: Upgrades

shell

cd /opt/litellm
/opt/litellm/backup.sh
docker compose pull
docker compose up -d
docker compose logs -f litellm

Always back up before pulling. LiteLLM applies schema migrations on start and rolling back after a failed migration requires the dump.

Troubleshooting

Container restarts repeatedly on first run. Check docker compose logs postgres. The most common cause is a stale ./postgres data directory from a previous attempt with a different password. Remove it and start fresh if there is no data worth keeping.

"Invalid master key" on every request. The value in .env and the value your client sends have drifted. Confirm with docker exec litellm env | grep MASTER. Restart with docker compose up -d after editing .env, since docker compose restart does not re-read the file.

Provider credentials fail after a redeploy. LITELLM_SALT_KEY changed. Any credential stored in the database is now undecryptable. Restore the original salt key from backup, or re-enter every provider key through the UI.

Fallbacks never trigger. Fallbacks fire on errors and rate limits, not on slow responses. Confirm the fallback model name matches a model_name in model_list exactly, and check that num_retries is not exhausting on the primary before the fallback is reached.

Streaming hangs at the client. proxy_buffering off is missing in Nginx, or an intermediate proxy is buffering.

Rate limits are not enforced consistently. With num_workers above 1, limits require Redis. Verify REDIS_HOST is set and check docker compose logs redis for connection activity.

Spend shows as zero. The model is not in LiteLLM's pricing map, which is common for local or custom deployments. Add explicit input_cost_per_token and output_cost_per_token under that model's litellm_params.

Cannot reach a local model on the host. From inside the container, localhost is the container itself. Use the Docker bridge gateway address, and confirm the host service is not bound to 127.0.0.1 only, since that will reject the bridge address.

Next Steps

Add Prometheus scraping of /metrics and build a Grafana dashboard for spend, latency, and error rate by model. Point a local vLLM or Ollama instance at the local-small entry to serve cheap requests from your own hardware with hosted fallback. If you run several applications, define a team per application so budget enforcement maps to how you actually account for cost.