LLM Provider Configuration

Understanding Agent Zero's Model Architecture

Agent Zero uses three types of models for different purposes:

This separation lets you allocate expensive, capable models where they matter most while using efficient models for routine operations.

Local LLMs with Ollama

Running models locally eliminates API costs and keeps all data on your server. Ollama makes local LLM deployment straightforward.

Hardware Considerations

Local inference requires significant RAM. The model must fit entirely in memory:

CPU inference is slower than GPU but entirely usable for async workflows. Expect 5-15 tokens/second on a modern CPU with 8B models, compared to 50+ tokens/second with a GPU. For most RamNode deployments, 8B parameter models hit the sweet spot.

Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

Verify the installation:

ollama --version

Ollama runs as a systemd service automatically:

sudo systemctl status ollama

Pull Models

Download models you want to use. Start with a capable general-purpose model:

# Recommended starting model - excellent balance of capability and size
ollama pull qwen2.5:7b

# Alternative options
ollama pull llama3.1:8b
ollama pull mistral:7b
ollama pull codellama:7b  # Specialized for code

For the utility model, a smaller variant works well:

ollama pull qwen2.5:3b
# or
ollama pull llama3.2:3b

Pull an embedding model:

ollama pull nomic-embed-text
# or
ollama pull mxbai-embed-large

List installed models:

ollama list

Test Local Inference

Verify models work before configuring Agent Zero:

ollama run qwen2.5:7b "Write a Python function that calculates factorial"

You should see the model generate a response. Exit with /bye or Ctrl+D. Check inference speed:

time ollama run qwen2.5:7b "What is 2+2?" --verbose

The --verbose flag shows tokens per second.

Configure Agent Zero for Ollama

Edit your .env file to use local models:

nano ~/agent-zero/.env

Add Ollama configuration:

# Ollama Configuration
API_URL_OLLAMA=http://localhost:11434

# Model Selection (use exact names from 'ollama list')
CHAT_MODEL_OLLAMA=qwen2.5:7b
UTILITY_MODEL_OLLAMA=qwen2.5:3b
EMBEDDING_MODEL_OLLAMA=nomic-embed-text

Set Ollama models as defaults:

# Default to local models
CHAT_MODEL_DEFAULT=qwen2.5:7b
UTILITY_MODEL_DEFAULT=qwen2.5:3b
EMBEDDING_MODEL_DEFAULT=nomic-embed-text

Ollama Performance Tuning

Increase Context Length

By default, Ollama uses 2048 token context. For Agent Zero's complex workflows, increase this:

# Create a custom model with larger context
ollama create qwen2.5-32k -f - <<EOF
FROM qwen2.5:7b
PARAMETER num_ctx 32768
EOF

Update your .env to use the custom model:

CHAT_MODEL_OLLAMA=qwen2.5-32k

Configure Memory Usage

Ollama automatically manages GPU/CPU memory, but you can tune behavior:

# Edit Ollama service configuration
sudo systemctl edit ollama

Add environment variables:

[Service]
Environment="OLLAMA_NUM_PARALLEL=2"
Environment="OLLAMA_MAX_LOADED_MODELS=2"

Restart Ollama:

sudo systemctl restart ollama

Keep Models Loaded

By default, Ollama unloads models after 5 minutes of inactivity. For responsive agents, keep models loaded:

# Set longer keep-alive (in .env or environment)
OLLAMA_KEEP_ALIVE=24h

Or load models persistently:

curl http://localhost:11434/api/generate -d '{"model": "qwen2.5:7b", "keep_alive": -1}'

Recommended Local Model Combinations

CHAT_MODEL_DEFAULT=qwen2.5:7b
UTILITY_MODEL_DEFAULT=qwen2.5:3b
EMBEDDING_MODEL_DEFAULT=nomic-embed-text

CHAT_MODEL_DEFAULT=qwen2.5:14b
UTILITY_MODEL_DEFAULT=qwen2.5:7b
EMBEDDING_MODEL_DEFAULT=mxbai-embed-large

CHAT_MODEL_DEFAULT=deepseek-coder:6.7b
UTILITY_MODEL_DEFAULT=qwen2.5:3b
EMBEDDING_MODEL_DEFAULT=nomic-embed-text

Hybrid Configurations

The most practical setup often combines cloud and local models—using local inference for routine tasks and cloud APIs for complex reasoning.

# Cloud for complex reasoning
CHAT_MODEL_DEFAULT=gpt-4o

# Local for utility tasks (no API cost)
UTILITY_MODEL_DEFAULT=qwen2.5:3b

# Local embeddings (runs constantly, saves significant cost)
EMBEDDING_MODEL_DEFAULT=nomic-embed-text

# Start with local
CHAT_MODEL_DEFAULT=qwen2.5:7b

# Configure cloud as available alternative
CHAT_MODEL_OPENAI=gpt-4o

# Primary
CHAT_MODEL_DEFAULT=gpt-4o
API_KEY_OPENAI=sk-...

# Backup
CHAT_MODEL_ANTHROPIC=claude-sonnet-4-20250514
API_KEY_ANTHROPIC=sk-ant-...

# Local fallback
CHAT_MODEL_OLLAMA=qwen2.5:7b

API Key Security

API keys grant access to paid services. Protect them:

Restrict File Permissions

chmod 600 ~/agent-zero/.env

This ensures only your user can read the file.

Use Environment Variables

For production, consider loading keys from environment variables rather than files:

# In ~/.bashrc or service file
export API_KEY_OPENAI="sk-..."

Then reference in .env:

API_KEY_OPENAI=${API_KEY_OPENAI}

Set Usage Limits

Most providers let you set spending caps:

• OpenAI: Settings → Limits → Set monthly budget
• Anthropic: Settings → Limits → Usage limits
• Google: Cloud Console → Budgets & Alerts

Set alerts at 50% and 80% of your budget to catch runaway usage.

Rotate Keys Periodically

Generate new API keys monthly and revoke old ones. This limits exposure if a key is compromised.

Testing Your Configuration

After configuring providers, restart Agent Zero and verify each model works:

# Restart if running as service
sudo systemctl restart agent-zero

# Or restart manually
cd ~/agent-zero
source venv/bin/activate
python run_ui.py

Test Prompts

Check logs for any model loading errors:

journalctl -u agent-zero -f
# And in another terminal:
journalctl -u ollama -f