Deploy Prefect on a VPS

Prefect is a Python-native workflow orchestrator built around the idea that your existing functions should become production pipelines with a decorator, not a rewrite. You mark a function with @flow, mark its steps with @task, and Prefect handles retries, scheduling, concurrency limits, caching, and observability without forcing your code into a DAG structure.

Prefect 3 splits cleanly into a server and one or more workers. The server holds state, schedules, and the UI. Workers poll work pools and execute flow runs. This guide deploys both on a single RamNode VPS with PostgreSQL behind the server, systemd managing the processes, and Nginx handling TLS.

Architecture Overview

• Prefect server. A FastAPI application serving the REST API and the UI. Backed by PostgreSQL.
• Work pool. A logical queue on the server. Deployments target a pool; workers subscribe to one.
• Worker. A long running process that polls its pool, picks up scheduled runs, and launches them. The process worker type runs flows as subprocesses on the same machine, which is what you want on a single VPS.
• Deployments. Server side records that pair a flow with a schedule, parameters, and a source location.

The server does not execute your code. It only stores state and hands runs to workers. That separation means you can restart the server mid-run without killing the run.

What You Will Need

• A RamNode VPS running Ubuntu 24.04 LTS with at least 4 GB RAM and 2 vCPU. The server alone runs in under 1 GB, but flow runs execute on the same box under the process worker. Size for your heaviest flow.
• Root or sudo access.
• A domain or subdomain with an A record pointing at the VPS.
• Python 3.11 or 3.12.

Step 1: Prepare the System

apt update && apt upgrade -y
apt install -y python3.12 python3.12-venv python3.12-dev build-essential \
  libpq-dev git curl pkg-config

Create the service user and directories.

useradd --system --create-home --home-dir /opt/prefect --shell /bin/bash prefect
mkdir -p /opt/prefect/{flows,storage,logs}
chown -R prefect:prefect /opt/prefect

Add swap on smaller plans.

fallocate -l 2G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab

Step 2: Install and Configure PostgreSQL

Prefect defaults to SQLite. SQLite works for local development and falls over as soon as the server, a worker, and several concurrent flow runs all write state at once. Use PostgreSQL.

apt install -y postgresql postgresql-contrib
systemctl enable --now postgresql

sudo -u postgres psql <<'SQL'
CREATE USER prefect WITH PASSWORD 'ReplaceWithAStrongPassword';
CREATE DATABASE prefect OWNER prefect;
GRANT ALL PRIVILEGES ON DATABASE prefect TO prefect;
SQL

Prefect writes a state record for every task run transition, which produces a lot of small writes. Tune /etc/postgresql/16/main/postgresql.conf:

shared_buffers = 512MB
work_mem = 16MB
maintenance_work_mem = 128MB
effective_cache_size = 2GB
max_connections = 100

systemctl restart postgresql

Step 3: Install Prefect

sudo -u prefect -i
python3.12 -m venv /opt/prefect/venv
source /opt/prefect/venv/bin/activate
pip install --upgrade pip setuptools wheel
pip install "prefect" asyncpg

The asyncpg driver is required. Prefect's database layer is fully async, and the standard psycopg2 driver will not work for the server connection string.

Confirm the version.

prefect version

Add integration libraries your flows need:

pip install prefect-aws prefect-dbt prefect-docker
pip install pandas requests sqlalchemy

Step 4: Configure the Server

Prefect reads configuration from environment variables or from a profile file at ~/.prefect/profiles.toml. For a systemd deployment, environment variables in an EnvironmentFile are cleaner because they apply consistently to the server, the worker, and any manual CLI use.

Create /opt/prefect/prefect.env:

PREFECT_HOME=/opt/prefect/.prefect
PREFECT_API_DATABASE_CONNECTION_URL=postgresql+asyncpg://prefect:ReplaceWithAStrongPassword@localhost:5432/prefect
PREFECT_API_URL=http://127.0.0.1:4200/api
PREFECT_UI_API_URL=https://prefect.example.com/api
PREFECT_SERVER_API_HOST=127.0.0.1
PREFECT_SERVER_API_PORT=4200
PREFECT_LOGGING_LEVEL=INFO
PREFECT_API_DATABASE_ECHO=false
PREFECT_SERVER_ANALYTICS_ENABLED=false
PREFECT_RESULTS_PERSIST_BY_DEFAULT=true
PREFECT_LOCAL_STORAGE_PATH=/opt/prefect/storage

PREFECT_UI_API_URL is the setting people most often get wrong. The UI is a browser application, so it needs the address your browser can reach, meaning the public HTTPS URL. PREFECT_API_URL is what server side processes and workers use, meaning loopback. Set them to the same value and either the UI or the worker will break.

Lock the file down since it holds the database password.

chown prefect:prefect /opt/prefect/prefect.env
chmod 600 /opt/prefect/prefect.env

Step 5: Enable Server Authentication

Prefect 3 self-hosted supports HTTP basic auth on the API. Turn it on. Without it, anyone who can reach the API can create deployments and run arbitrary code.

Add to /opt/prefect/prefect.env:

PREFECT_SERVER_API_AUTH_STRING=admin:ReplaceWithAStrongUIPassword
PREFECT_API_AUTH_STRING=admin:ReplaceWithAStrongUIPassword

PREFECT_SERVER_API_AUTH_STRING is what the server enforces. PREFECT_API_AUTH_STRING is what clients, including your worker and CLI, present. They must match.

This is server-wide basic auth, not per-user accounts. There is no user management or RBAC in the open source server. If you need real multi-user access control, front the deployment with an identity aware proxy such as Authelia or oauth2-proxy and keep basic auth as a second layer.

Step 6: Initialize the Database

Run the migrations before starting the service.

sudo -u prefect -i
set -a && source /opt/prefect/prefect.env && set +a
source /opt/prefect/venv/bin/activate
prefect server database upgrade -y

Confirm the tables exist.

sudo -u postgres psql -d prefect -c '\dt' | head -20

Step 7: Create the Server systemd Service

Exit to root. Create /etc/systemd/system/prefect-server.service:

[Unit]
Description=Prefect Server
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=prefect
Group=prefect
WorkingDirectory=/opt/prefect
EnvironmentFile=/opt/prefect/prefect.env
ExecStart=/opt/prefect/venv/bin/prefect server start \
  --host 127.0.0.1 \
  --port 4200
Restart=on-failure
RestartSec=10
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable --now prefect-server
systemctl status prefect-server --no-pager

Check that it is listening.

curl -u admin:ReplaceWithAStrongUIPassword http://127.0.0.1:4200/api/health

A healthy server returns true.

Step 8: Create a Work Pool and Worker

Work pools are created against the running server.

sudo -u prefect -i
set -a && source /opt/prefect/prefect.env && set +a
source /opt/prefect/venv/bin/activate
prefect work-pool create --type process default-process-pool
prefect work-pool ls

Set a concurrency limit on the pool so a backfill cannot spawn fifty subprocesses and take the VPS down.

prefect work-pool set-concurrency-limit default-process-pool 4

Create the worker service at /etc/systemd/system/prefect-worker.service:

[Unit]
Description=Prefect Process Worker
After=network.target prefect-server.service
Requires=prefect-server.service

[Service]
Type=simple
User=prefect
Group=prefect
WorkingDirectory=/opt/prefect/flows
EnvironmentFile=/opt/prefect/prefect.env
ExecStart=/opt/prefect/venv/bin/prefect worker start \
  --pool default-process-pool \
  --name ramnode-worker-1 \
  --limit 4
Restart=on-failure
RestartSec=10
TimeoutStopSec=60
KillMode=mixed

[Install]
WantedBy=multi-user.target

KillMode=mixed sends SIGTERM to the worker but lets it propagate shutdown to its child flow processes rather than having systemd kill them all at once. Without it, a systemctl restart leaves orphaned flow runs stuck in RUNNING.

systemctl daemon-reload
systemctl enable --now prefect-worker
systemctl status prefect-worker --no-pager

Step 9: Configure Nginx and TLS

apt install -y nginx certbot python3-certbot-nginx

Create /etc/nginx/sites-available/prefect:

server {
    listen 80;
    server_name prefect.example.com;

    client_max_body_size 50M;

    location / {
        proxy_pass http://127.0.0.1:4200;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_read_timeout 600;
        proxy_send_timeout 600;
        proxy_buffering off;
    }
}

Prefect uses server sent events for live UI updates. proxy_buffering off is what keeps the flow run view updating in real time instead of freezing until a run completes.

ln -s /etc/nginx/sites-available/prefect /etc/nginx/sites-enabled/
rm -f /etc/nginx/sites-enabled/default
nginx -t
systemctl reload nginx
certbot --nginx -d prefect.example.com

Firewall:

ufw allow OpenSSH
ufw allow 'Nginx Full'
ufw --force enable

Verify the API is not exposed directly.

ss -tlnp | grep 4200

It should show 127.0.0.1:4200.

Step 10: Write and Deploy Your First Flow

Create /opt/prefect/flows/etl.py:

import httpx
from prefect import flow, task, get_run_logger
from prefect.tasks import task_input_hash
from datetime import timedelta


@task(retries=3, retry_delay_seconds=10, cache_key_fn=task_input_hash,
      cache_expiration=timedelta(hours=1))
def fetch_data(url: str) -> dict:
    logger = get_run_logger()
    response = httpx.get(url, timeout=30)
    response.raise_for_status()
    logger.info(f"Fetched {len(response.content)} bytes from {url}")
    return response.json()


@task
def transform(payload: dict) -> int:
    logger = get_run_logger()
    count = len(payload) if isinstance(payload, (list, dict)) else 0
    logger.info(f"Transformed {count} records")
    return count


@flow(name="daily-etl", log_prints=True)
def daily_etl(url: str = "https://httpbin.org/json"):
    raw = fetch_data(url)
    count = transform(raw)
    print(f"Pipeline complete, processed {count} items")
    return count


if __name__ == "__main__":
    daily_etl()

Test it directly before deploying.

sudo -u prefect -i
set -a && source /opt/prefect/prefect.env && set +a
source /opt/prefect/venv/bin/activate
cd /opt/prefect/flows
python etl.py

Now create a deployment. Write /opt/prefect/flows/prefect.yaml:

name: ramnode-flows
prefect-version: 3.0.0

pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /opt/prefect/flows

deployments:
  - name: daily-etl-prod
    entrypoint: etl.py:daily_etl
    work_pool:
      name: default-process-pool
    schedules:
      - cron: "0 6 * * *"
        timezone: "America/Chicago"
    parameters:
      url: "https://httpbin.org/json"
    tags:
      - production
    description: "Daily ETL pipeline"

Apply it.

cd /opt/prefect/flows
prefect deploy --all

Set the timezone explicitly in every schedule. Prefect defaults to UTC, and a cron entry of 0 6 * * * firing at midnight local time is a surprise most people only discover once.

Trigger a run manually to confirm the worker picks it up.

prefect deployment run 'daily-etl/daily-etl-prod'

Watch the worker.

journalctl -u prefect-worker -f

Step 11: Deploying from Git

For real work, pull flow code from a repository rather than editing files on the server. Replace the pull step in prefect.yaml:

pull:
  - prefect.deployments.steps.git_clone:
      repository: https://github.com/youruser/your-flows.git
      branch: main
      credentials: "{{ prefect.blocks.github-credentials.repo-creds }}"

Create the credentials block first:

prefect block register -m prefect_github

Then add the block through the UI under Blocks, or in Python:

from prefect_github import GitHubCredentials

GitHubCredentials(token="ghp_yourtoken").save("repo-creds")

With git_clone, every flow run clones the repo fresh into a temporary directory. That gives you reproducible runs and removes the need to redeploy code manually, but it does mean the worker's virtual environment must already contain every dependency your flows import.

Verifying the Deployment

Load the UI at your domain. Basic auth prompts once, then you land on the dashboard.

Check that the worker is registered and polling:

prefect work-pool inspect default-process-pool

The pool status should show at least one online worker with a recent heartbeat.

Confirm the schedule is active in the UI under Deployments. A deployment with a schedule that shows as paused will never run.

Troubleshooting

UI loads but shows no data and the browser console shows failed API calls. PREFECT_UI_API_URL is set to the loopback address. Set it to your public HTTPS URL plus /api and restart the server.

Worker starts then immediately exits with a connection error. PREFECT_API_URL is wrong, or PREFECT_API_AUTH_STRING does not match PREFECT_SERVER_API_AUTH_STRING. Check both in the environment file.

Flow runs sit in Scheduled and never start. No worker is polling that pool, or the pool concurrency limit is saturated. Run prefect work-pool inspect and check for a stale worker heartbeat.

Flow runs go straight to Crashed. The worker cannot import your flow module. With set_working_directory, confirm the path is correct and the prefect user can read it. With git_clone, check the worker logs for clone failures.

"asyncpg is required" on server start. The connection URL uses postgresql+asyncpg:// but the package is not installed in the venv, or the URL is missing the +asyncpg dialect suffix.

Runs stuck in Running after a restart. The worker was killed without propagating shutdown. Add KillMode=mixed to the worker unit. Clear the stuck runs from the UI, or set a flow run timeout so they self-terminate.

Database connections exhausted. Prefect holds a connection pool per process. If you run multiple workers, raise max_connections in PostgreSQL or lower PREFECT_API_DATABASE_CONNECTION_POOL_SIZE.

Scaling on One VPS

You can run more than one worker against the same pool, or create separate pools for different workload classes.

prefect work-pool create --type process heavy-pool
prefect work-pool set-concurrency-limit heavy-pool 1

Then copy the worker unit, change the pool name and worker name, and enable it. Route memory hungry deployments at heavy-pool and everything else at the default pool. That prevents one large job from starving your quick flows.

Global concurrency limits give you finer control, for instance capping calls to a rate limited API across all flows:

prefect gcl create api-calls --limit 5

Then use it in a flow:

from prefect.concurrency.sync import concurrency

with concurrency("api-calls", occupy=1):
    result = call_external_api()

Maintenance

Prefect's state tables grow with every task run. Set retention on your deployments and clean old runs periodically. There is no built in retention policy in the open source server, so schedule a Prefect flow that calls the API to delete old flow runs:

from prefect import flow, get_client
from prefect.client.schemas.filters import FlowRunFilter, FlowRunFilterStartTime
from datetime import datetime, timedelta, timezone


@flow
async def prune_runs(days: int = 30):
    cutoff = datetime.now(timezone.utc) - timedelta(days=days)
    async with get_client() as client:
        runs = await client.read_flow_runs(
            flow_run_filter=FlowRunFilter(
                start_time=FlowRunFilterStartTime(before_=cutoff)
            ),
            limit=500,
        )
        for run in runs:
            await client.delete_flow_run(run.id)
        return len(runs)

Deploy that on a weekly schedule.

Back up the database:

sudo -u postgres pg_dump prefect | gzip > /root/prefect-$(date +%F).sql.gz

Upgrade with services stopped:

systemctl stop prefect-worker prefect-server
sudo -u prefect /opt/prefect/venv/bin/pip install --upgrade prefect
sudo -u prefect -i
set -a && source /opt/prefect/prefect.env && set +a
source /opt/prefect/venv/bin/activate
prefect server database upgrade -y
exit
systemctl start prefect-server prefect-worker

Always run the database upgrade after a version bump. Prefect will refuse to start against a schema it does not recognize.

Hardening Notes

• The worker executes arbitrary Python from your deployments. Never give the prefect user sudo.
• Store secrets in Prefect Secret blocks or the systemd environment file, never in flow source.
• Restrict PostgreSQL to loopback in pg_hba.conf.
• Rotate PREFECT_SERVER_API_AUTH_STRING on a schedule and remember that every worker and CLI client must be updated at the same time.
• Set flow run timeouts with @flow(timeout_seconds=3600) so a hung run cannot hold a concurrency slot indefinitely.
• If you expose the API to workers on other hosts, keep basic auth on and restrict the source IPs in ufw rather than opening port 4200 to the world.

Where to Go Next

Prefect on a single VPS handles a surprising amount of production work, particularly when the heavy compute happens elsewhere and the flows are mostly coordination. When you need isolation between flows, switch the pool type from process to docker and each run gets its own container with its own dependency set. Pair this deployment with a BI layer such as Apache Superset reading the tables your flows produce.