Monitoring Guide

    Deploy Thanos Long-Term Metrics

    Thanos extends Prometheus with unlimited retention, global query views, and downsampling — all without replacing the tools you already use. Deploy it on RamNode's reliable VPS hosting with MinIO as your self-hosted object store.

    Unlimited Retention
    Global Query
    Downsampling
    S3 Object Storage
    1

    Introduction

    Prometheus is the gold standard for metrics collection, but it was never designed for long-term retention. By default, data is stored locally and purged after 15 days. That's fine for short-term alerting, but useless when you want to analyze trends across weeks or months.

    Thanos solves this by layering on top of your existing Prometheus setup, federating metrics across multiple instances, and offloading historical data to object storage. The result: unlimited retention, global query views, and downsampling for faster long-range queries.

    2

    Architecture Overview

    Component topology
    Prometheus (scraping targets)
        |
        +-- Thanos Sidecar (reads TSDB blocks, exposes gRPC Store API)
                    |
                    +-- Object Storage (MinIO on-disk)
                                |
                                +-- Thanos Store Gateway (serves historical blocks)
                                +-- Thanos Compactor (downsamples + deduplicates)
                    |
        Thanos Querier (merges live + historical data)
                    |
        Grafana (visualization)

    The Sidecar runs alongside Prometheus and ships completed TSDB blocks to object storage every two hours. The Store Gateway makes those blocks queryable. The Querier fans out requests across both the Sidecar (recent data) and Store Gateway (historical data), giving you a unified view.

    3

    Prerequisites

    • A RamNode KVM VPS running Ubuntu 22.04 (4GB RAM minimum; 8GB recommended)
    • Prometheus already installed and running
    • Docker and Docker Compose installed
    • A domain or static IP for your VPS
    • Ports 9090, 10901–10907, 9000, and 9001 available

    If Prometheus is not yet installed, see our Prometheus Getting Started guide first.

    4

    Reconfigure Prometheus for Thanos

    Thanos requires Prometheus to store data in a location the Sidecar can read. You also need to set block durations so Thanos, not Prometheus, handles compaction.

    Edit Prometheus systemd service
    sudo nano /etc/systemd/system/prometheus.service
    Update ExecStart line
    ExecStart=/usr/local/bin/prometheus \
      --config.file=/etc/prometheus/prometheus.yml \
      --storage.tsdb.path=/var/lib/prometheus/data \
      --storage.tsdb.min-block-duration=2h \
      --storage.tsdb.max-block-duration=2h \
      --storage.tsdb.retention.time=6h \
      --web.enable-lifecycle \
      --web.listen-address=0.0.0.0:9090

    Key points: Setting both block durations to 2h disables Prometheus's own compaction — Thanos Compactor handles this from object storage. Retention of 6h keeps only recent data in Prometheus; Thanos serves everything older.

    Reload and restart
    sudo systemctl daemon-reload
    sudo systemctl restart prometheus
    sudo systemctl status prometheus
    5

    Create Object Storage with MinIO

    MinIO gives you an S3-compatible store running entirely on your RamNode VPS. All metrics blocks will be written here.

    Create working directory
    sudo mkdir -p /opt/thanos
    cd /opt/thanos
    /opt/thanos/docker-compose.yml (MinIO)
    version: "3.8"
    
    services:
      minio:
        image: quay.io/minio/minio:latest
        container_name: minio
        restart: unless-stopped
        ports:
          - "9000:9000"
          - "9001:9001"
        environment:
          MINIO_ROOT_USER: thanosadmin
          MINIO_ROOT_PASSWORD: changeme-use-a-strong-password
        volumes:
          - /opt/thanos/minio-data:/data
        command: server /data --console-address ":9001"
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
          interval: 30s
          timeout: 10s
          retries: 3

    Security note: Change MINIO_ROOT_PASSWORD before going to production. Do not expose port 9000 or 9001 publicly without authentication.

    Start MinIO and create bucket
    sudo docker compose up -d minio
    
    # Install MinIO client if needed
    curl -sSL https://dl.min.io/client/mc/release/linux-amd64/mc -o /usr/local/bin/mc
    chmod +x /usr/local/bin/mc
    
    # Configure alias and create bucket
    mc alias set local http://localhost:9000 thanosadmin changeme-use-a-strong-password
    mc mb local/thanos-metrics
    mc ls local/
    6

    Create Object Store Configuration

    Thanos uses a YAML config file to authenticate with your object store.

    /etc/thanos-objstore.yml
    type: S3
    config:
      bucket: thanos-metrics
      endpoint: localhost:9000
      access_key: thanosadmin
      secret_key: changeme-use-a-strong-password
      insecure: true
    Set restrictive permissions
    sudo chmod 600 /etc/thanos-objstore.yml
    7

    Deploy the Thanos Sidecar

    The Sidecar runs next to Prometheus, uploads TSDB blocks to object storage, and exposes a gRPC Store API endpoint so the Querier can read recent data directly from Prometheus.

    Append to docker-compose.yml
      thanos-sidecar:
        image: quay.io/thanos/thanos:v0.35.1
        container_name: thanos-sidecar
        restart: unless-stopped
        network_mode: host
        volumes:
          - /var/lib/prometheus/data:/prometheus-data
          - /etc/thanos-objstore.yml:/etc/thanos-objstore.yml:ro
        command:
          - sidecar
          - --tsdb.path=/prometheus-data
          - --prometheus.url=http://localhost:9090
          - --objstore.config-file=/etc/thanos-objstore.yml
          - --http-address=0.0.0.0:10902
          - --grpc-address=0.0.0.0:10901

    The Sidecar uses network_mode: host so it can reach Prometheus on localhost:9090. Adjust if you run Prometheus in Docker as well.

    Start the Sidecar
    cd /opt/thanos
    sudo docker compose up -d thanos-sidecar
    sudo docker compose logs -f thanos-sidecar

    Wait for a log line like Starting shipper. The first block upload happens after two hours.

    8

    Deploy the Store Gateway

    The Store Gateway reads blocks from MinIO and exposes them via the same gRPC Store API. This gives the Querier access to historical data.

    Append to docker-compose.yml
      thanos-store:
        image: quay.io/thanos/thanos:v0.35.1
        container_name: thanos-store
        restart: unless-stopped
        ports:
          - "10903:10903"
          - "10904:10904"
        volumes:
          - /opt/thanos/store-data:/store-data
          - /etc/thanos-objstore.yml:/etc/thanos-objstore.yml:ro
        command:
          - store
          - --objstore.config-file=/etc/thanos-objstore.yml
          - --data-dir=/store-data
          - --http-address=0.0.0.0:10904
          - --grpc-address=0.0.0.0:10903
    Create data directory and start
    sudo mkdir -p /opt/thanos/store-data
    sudo docker compose up -d thanos-store
    9

    Deploy the Compactor

    The Compactor downsamples blocks for faster long-range queries and deduplicates data from multiple Prometheus replicas. Only run one Compactor per bucket.

    Append to docker-compose.yml
      thanos-compactor:
        image: quay.io/thanos/thanos:v0.35.1
        container_name: thanos-compactor
        restart: unless-stopped
        ports:
          - "10905:10905"
        volumes:
          - /opt/thanos/compactor-data:/compactor-data
          - /etc/thanos-objstore.yml:/etc/thanos-objstore.yml:ro
        command:
          - compact
          - --objstore.config-file=/etc/thanos-objstore.yml
          - --data-dir=/compactor-data
          - --http-address=0.0.0.0:10905
          - --retention.resolution-raw=30d
          - --retention.resolution-5m=90d
          - --retention.resolution-1h=365d
          - --wait

    Retention Policy

    ResolutionRetention
    Raw (15s/30s scrape)30 days
    5-minute downsamples90 days
    1-hour downsamples1 year
    Create data directory and start
    sudo mkdir -p /opt/thanos/compactor-data
    sudo docker compose up -d thanos-compactor
    10

    Deploy the Querier

    The Querier is the component Grafana talks to. It fans out PromQL queries across all configured Store API endpoints and deduplicates overlapping data.

    Append to docker-compose.yml
      thanos-querier:
        image: quay.io/thanos/thanos:v0.35.1
        container_name: thanos-querier
        restart: unless-stopped
        ports:
          - "10906:10906"
          - "10907:10907"
        command:
          - query
          - --http-address=0.0.0.0:10906
          - --grpc-address=0.0.0.0:10907
          - --store=host.docker.internal:10901
          - --store=thanos-store:10903
          - --query.replica-label=prometheus_replica
        extra_hosts:
          - "host.docker.internal:host-gateway"

    The --store flags point to the Sidecar (host network) and the Store Gateway container.

    Start the Querier
    sudo docker compose up -d thanos-querier

    Verify at http://your-vps-ip:10906. Navigate to Stores and confirm both endpoints show as UP.

    11

    Connect Grafana

    Add a new Prometheus data source in Grafana pointing to the Thanos Querier instead of Prometheus directly:

    SettingValue
    URLhttp://your-vps-ip:10906
    TypePrometheus

    This is the only change needed. All existing dashboards continue to work, but long-range queries now transparently pull from object storage.

    12

    Verify the Full Pipeline

    Sanity checks
    # Check all containers are running
    sudo docker compose -f /opt/thanos/docker-compose.yml ps
    
    # Query via the Thanos HTTP API for a known metric
    curl -s "http://localhost:10906/api/v1/query?query=up" | python3 -m json.tool | head -40
    
    # Check block uploads in MinIO
    mc ls local/thanos-metrics/

    After the first two-hour TSDB block cycle completes, you should see block directories appear in local/thanos-metrics/. The Store Gateway will index them automatically.

    13

    Resource Expectations

    On a 4-vCPU / 8GB RAM RamNode KVM VPS with roughly 50 scrape targets at 15s intervals:

    ComponentRAMCPU (idle)
    Prometheus~600 MBLow
    Thanos Sidecar~80 MBLow
    Thanos Store Gateway~200 MBLow (spikes on query)
    Thanos Querier~100 MBLow (spikes on query)
    Thanos Compactor~300 MBPeriodic bursts
    MinIO~150 MBLow

    Total: roughly 1.5 GB RAM at steady state, leaving plenty of headroom on an 8GB node. On a 4GB VPS, consider running the Compactor on a schedule rather than with --wait.

    Troubleshooting

    IssueSolution
    Sidecar can't connect to PrometheusConfirm Prometheus listens on 0.0.0.0:9090, not 127.0.0.1
    Store endpoints show as DOWNCheck firewall: sudo ufw allow from 172.16.0.0/12 to any port 10901
    MinIO bucket not foundRe-run mc mb local/thanos-metrics and verify name in objstore config
    Compactor: "too many blocks"Normal on first run — let it work through the backlog with --wait
    Disk space growing quicklyTighten --retention.resolution-raw or reduce scrape frequency

    Thanos Long-Term Metrics Deployed!

    You now have unlimited Prometheus metrics retention with Thanos on your RamNode VPS. Next steps to consider:

    • Add a second Prometheus replica for HA — the Compactor handles deduplication automatically
    • Enable TLS on gRPC endpoints if accessible outside your private network
    • Set up Thanos Ruler for recording rules and alerts over long-range data
    • Monitor Thanos itself — each component exposes /metrics on its HTTP port