Kubernetes Backup Guide

    Velero Kubernetes Backup Guide

    Velero is an open-source tool for backing up and restoring Kubernetes cluster resources and persistent volumes. Combined with RamNode's S3-compatible object storage, you can implement robust disaster recovery for your clusters.

    Kubernetes Clusters
    Disaster Recovery
    ⏱️ 30-45 minutes

    Introduction

    Velero (formerly known as Heptio Ark) is a backup and disaster recovery solution specifically designed for Kubernetes clusters. It allows you to back up cluster resources, restore on demand, and migrate workloads between clusters.

    Cluster Resources

    Capture the current state of your Kubernetes resources

    Restore on Demand

    Quickly recover from disasters or migrate workloads

    Scheduled Backups

    Set up recurring backups for peace of mind

    Multi-Cloud Support

    AWS, GCP, Azure, and S3-compatible storage

    1

    Prerequisites

    Hardware Requirements

    • RamNode VPS: Minimum 2 CPU cores, 4GB RAM, 40GB SSD storage
    • Kubernetes Cluster: Running Kubernetes 1.16 or later
    • Storage: Additional space for backup storage

    Software Requirements

    • Ubuntu 22.04 LTS (or similar Linux distribution)
    • Kubernetes cluster (k3s, kubeadm, or managed Kubernetes)
    • kubectl configured and connected to your cluster
    • Helm 3.x (optional, for Helm-based installation)
    • Object storage provider (MinIO, AWS S3, Backblaze B2, etc.)
    2

    Setting Up Your RamNode VPS

    Initial Server Setup

    Update system and install tools
    # Update system packages
    sudo apt update && sudo apt upgrade -y
    
    # Install essential tools
    sudo apt install -y curl wget git vim

    Install kubectl

    Install kubectl
    # Download latest kubectl
    curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
    
    # Install kubectl
    sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
    
    # Verify installation
    kubectl version --client

    Verify Kubernetes Cluster

    Check cluster status
    # Check cluster status
    kubectl cluster-info
    
    # View nodes
    kubectl get nodes
    
    # Check system pods
    kubectl get pods -A
    3

    Setting Up Object Storage

    Velero requires object storage to store backups. You can use MinIO (self-hosted) or an external S3-compatible service.

    Option A: Deploy MinIO on Kubernetes

    Create MinIO namespace
    kubectl create namespace minio
    Deploy MinIO
    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: minio-pvc
      namespace: minio
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: minio
      namespace: minio
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: minio
      template:
        metadata:
          labels:
            app: minio
        spec:
          containers:
          - name: minio
            image: minio/minio:latest
            args:
            - server
            - /data
            - --console-address
            - ":9001"
            env:
            - name: MINIO_ROOT_USER
              value: "minio"
            - name: MINIO_ROOT_PASSWORD
              value: "minio123"
            ports:
            - containerPort: 9000
            - containerPort: 9001
            volumeMounts:
            - name: storage
              mountPath: /data
          volumes:
          - name: storage
            persistentVolumeClaim:
              claimName: minio-pvc
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: minio
      namespace: minio
    spec:
      type: ClusterIP
      ports:
      - port: 9000
        targetPort: 9000
        name: api
      - port: 9001
        targetPort: 9001
        name: console
      selector:
        app: minio
    EOF

    Security Note

    In production, use strong passwords and secure MinIO with proper authentication and TLS certificates.

    Create MinIO Bucket for Velero

    Create bucket
    # Port forward to access MinIO
    kubectl port-forward -n minio svc/minio 9000:9000 &
    
    # Install MinIO client
    wget https://dl.min.io/client/mc/release/linux-amd64/mc
    chmod +x mc
    sudo mv mc /usr/local/bin/
    
    # Configure MinIO client
    mc alias set minio http://localhost:9000 minio minio123
    
    # Create bucket for Velero
    mc mb minio/velero-backups
    
    # Verify bucket creation
    mc ls minio/
    4

    Installing Velero

    Download Velero CLI

    Install Velero CLI
    # Download latest Velero release
    VELERO_VERSION="v1.12.3"
    wget https://github.com/vmware-tanzu/velero/releases/download/${VELERO_VERSION}/velero-${VELERO_VERSION}-linux-amd64.tar.gz
    
    # Extract and install
    tar -xvf velero-${VELERO_VERSION}-linux-amd64.tar.gz
    sudo mv velero-${VELERO_VERSION}-linux-amd64/velero /usr/local/bin/
    rm -rf velero-${VELERO_VERSION}-linux-amd64*
    
    # Verify installation
    velero version --client-only

    Create Credentials File

    For MinIO
    cat <<EOF > /tmp/credentials-velero
    [default]
    aws_access_key_id = minio
    aws_secret_access_key = minio123
    EOF

    Install Velero in Your Cluster

    Install Velero with MinIO
    velero install \
        --provider aws \
        --plugins velero/velero-plugin-for-aws:v1.8.2 \
        --bucket velero-backups \
        --secret-file /tmp/credentials-velero \
        --use-volume-snapshots=false \
        --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.minio.svc.cluster.local:9000

    Verify Installation

    Check Velero status
    # Check Velero namespace
    kubectl get namespace velero
    
    # Check Velero pods
    kubectl get pods -n velero
    
    # Check backup storage location
    velero backup-location get
    
    # Verify Velero is ready
    kubectl get deployment velero -n velero
    Clean up credentials
    rm /tmp/credentials-velero
    5

    Creating Your First Backup

    Deploy a Test Application

    Create test deployment
    # Create test namespace
    kubectl create namespace backup-test
    
    # Deploy nginx
    kubectl create deployment nginx --image=nginx:latest -n backup-test
    
    # Expose nginx
    kubectl expose deployment nginx --port=80 --type=ClusterIP -n backup-test
    
    # Create a ConfigMap with some data
    kubectl create configmap nginx-config \
        --from-literal=index.html="Hello from Velero Backup Test!" \
        -n backup-test
    
    # Verify deployment
    kubectl get all -n backup-test

    Create a Backup

    Backup the namespace
    # Create backup
    velero backup create backup-test-01 --include-namespaces backup-test
    
    # Check backup status
    velero backup describe backup-test-01
    
    # Get backup details
    velero backup get

    Monitor Backup Progress

    Check backup logs
    # Watch backup progress
    velero backup logs backup-test-01
    
    # Check backup completion
    velero backup describe backup-test-01 --details
    6

    Testing Restore

    Delete the Test Application

    Delete namespace
    # Delete the entire namespace
    kubectl delete namespace backup-test
    
    # Verify deletion
    kubectl get namespace backup-test

    Restore from Backup

    Restore backup
    # Create restore from backup
    velero restore create --from-backup backup-test-01
    
    # Check restore status
    velero restore get
    
    # Describe restore details
    velero restore describe backup-test-01-<timestamp> --details

    Verify Restoration

    Verify restored resources
    # Check if namespace was restored
    kubectl get namespace backup-test
    
    # Verify all resources
    kubectl get all -n backup-test
    
    # Check ConfigMap
    kubectl get configmap nginx-config -n backup-test -o yaml
    7

    Setting Up Scheduled Backups

    Create a Daily Backup Schedule

    Daily backup schedule
    # Create schedule for daily backups at 2 AM UTC
    velero schedule create daily-backup \
        --schedule="0 2 * * *" \
        --include-namespaces "*" \
        --exclude-namespaces kube-system,kube-public,kube-node-lease,velero \
        --ttl 168h0m0s

    This creates a schedule that:

    • Runs daily at 2:00 AM UTC
    • Backs up all namespaces except system namespaces
    • Keeps backups for 7 days (168 hours)

    Namespace-Specific Schedules

    Custom schedules
    # Backup production namespace every 6 hours
    velero schedule create production-backup \
        --schedule="0 */6 * * *" \
        --include-namespaces production \
        --ttl 72h0m0s
    
    # Backup databases more frequently (every 4 hours)
    velero schedule create database-backup \
        --schedule="0 */4 * * *" \
        --selector app=database \
        --ttl 48h0m0s

    Manage Schedules

    Schedule management
    # List all schedules
    velero schedule get
    
    # Describe a schedule
    velero schedule describe daily-backup
    
    # Pause a schedule
    velero schedule pause daily-backup
    
    # Resume a schedule
    velero schedule unpause daily-backup
    
    # Delete a schedule
    velero schedule delete daily-backup
    8

    Advanced Backup Strategies

    Selective Backups with Labels

    Label-based backups
    # Backup only resources with specific label
    velero backup create critical-apps \
        --selector app=critical \
        --include-namespaces production
    
    # Exclude resources with specific label
    velero backup create non-critical \
        --selector app!=critical

    Backup with Resource Filters

    Resource filtering
    # Backup specific resources only
    velero backup create config-backup \
        --include-resources configmaps,secrets \
        --include-namespaces production
    9

    Monitoring and Maintenance

    Check Backup Location Status

    Verify storage
    # Verify backup storage location
    velero backup-location get
    
    # Get detailed information
    velero backup-location describe default

    Monitor Backup Statistics

    Backup monitoring
    # List all backups
    velero backup get
    
    # Check failed backups
    velero backup get --selector velero.io/backup-status=Failed
    
    # View backup logs
    velero backup logs <backup-name>

    Prometheus Metrics

    Access metrics
    # Check Velero metrics endpoint
    kubectl port-forward -n velero deployment/velero 8085:8085 &
    curl http://localhost:8085/metrics
    10

    Disaster Recovery Procedures

    Full Cluster Restore

    In case of complete cluster failure:

    1. Rebuild Kubernetes cluster on new RamNode VPS
    2. Reinstall Velero with same configuration
    3. Restore all backups
    Full restore
    velero restore create full-restore \
        --from-backup daily-backup-<timestamp> \
        --restore-volumes=true
    
    # Monitor restore
    velero restore describe full-restore --details

    Partial Restore

    Selective restore
    # Restore only deployments
    velero restore create selective-restore \
        --from-backup backup-name \
        --include-resources deployments \
        --include-namespaces production

    Troubleshooting

    Backup Stuck in "InProgress" Status

    # Check Velero logs
    kubectl logs deployment/velero -n velero
    
    # Check if backup location is accessible
    velero backup-location get
    
    # Restart Velero
    kubectl rollout restart deployment/velero -n velero

    BackupStorageLocation Unavailable

    # Verify storage credentials
    kubectl get secret -n velero cloud-credentials -o yaml
    
    # Update backup location
    velero backup-location set default --access-mode=ReadWrite

    Restore Fails with Resource Conflicts

    # Use --existing-resource-policy to handle conflicts
    velero restore create restore-with-update \
        --from-backup backup-name \
        --existing-resource-policy=update
    
    # Or skip existing resources
    velero restore create restore-skip-existing \
        --from-backup backup-name \
        --existing-resource-policy=none

    Best Practices

    Follow 3-2-1 Rule

    Keep 3 copies of data, on 2 different media, with 1 copy offsite.

    Test Restores Regularly

    Schedule monthly restore tests to verify backups work.

    Multiple Backup Locations

    Configure multiple backup storage locations for redundancy.

    Document Procedures

    Create runbooks for backup and restore procedures.

    You're All Set!

    You now have a fully functional Velero deployment on your RamNode VPS, providing enterprise-grade backup and disaster recovery capabilities for your Kubernetes workloads.

    Key takeaways:

    • Velero provides automated backup and restore for Kubernetes clusters
    • Object storage (MinIO, S3, B2) is required for storing backups
    • Scheduled backups ensure continuous data protection
    • Regular testing of restore procedures is critical