Recovering from a Full Root Partition

When the root filesystem on a Linux VPS reaches 100% disk usage, the system enters a failure cascade that blocks most normal recovery paths:

• SSH connections are refused or immediately drop because sshd cannot write to /var/run or /tmp
• New processes cannot fork because the kernel has no room to write PID files
• Log rotation fails, making the problem compound silently
• Package managers lock up mid-operation, leaving broken state
• Databases (MySQL, PostgreSQL, Redis) crash or refuse writes

The result is a server that appears online from outside (ping responds, port 22 may even accept the TCP connection) but immediately terminates every session attempt.

The RamNode client area provides two out-of-band access methods that bypass the broken SSH stack entirely: the VNC Console and Rescue Mode. This guide covers both, in order of speed and simplicity.

Log in to the RamNode client area at my.ramnode.com. Navigate to Services → My Services and click the VPS that is down. From the VPS management page you have two tools available.

The VNC console connects directly to the virtual machine's video output and keyboard, bypassing the network stack completely. The server does not need a working SSH daemon, open firewall ports, or even a healthy network configuration.

1. In the RamNode client area, open your VPS management page.
2. Click the Console button. A browser-based noVNC window opens.
3. If the screen is black, press Enter or a letter key to wake the display.
4. You will see either a login prompt, a kernel panic screen, or a frozen shell.
5. Log in with your root credentials.

Rescue Mode boots a minimal, self-contained Linux environment that mounts your VPS disk as a secondary device. Your root partition is not the active OS, so you can delete files, run fsck, or resize the filesystem without lock contention.

1. From the VPS management page, click Rescue Mode (or Boot into Rescue).
2. Choose a rescue distribution if prompted. Debian or Ubuntu minimal are recommended.
3. A temporary root password will be displayed. Copy it immediately — it is only shown once.
4. Wait 60–90 seconds for the rescue kernel to boot.
5. Connect via the VNC console using the temporary password.
6. Your production disk will be visible as a block device, typically /dev/vda1 or /dev/sda1.

# Inside rescue mode -- find your production partition
lsblk

# Mount it (adjust device path from lsblk output)
mkdir -p /mnt/recovery
mount /dev/vda1 /mnt/recovery

# Verify the mount and check disk usage
df -h /mnt/recovery
du -sh /mnt/recovery/* | sort -rh | head -20

If you can log in via the VNC console (your OS is still running, just SSH is broken), work directly in the live system. The goal is to free enough space to restore SSH access, then do a full cleanup after.

# Confirm the partition is full
df -h /

# Find the top space consumers in /var (most common culprit)
du -sh /var/* 2>/dev/null | sort -rh | head -20

# Find the top space consumers across the entire filesystem
du -sh /* 2>/dev/null | sort -rh | head -20

# Find individual large files over 100MB
find / -xdev -type f -size +100M -printf '%s	%p
' 2>/dev/null | sort -rn | head -20

The following file categories are safe to delete without rebooting or affecting running services. Focus on these first.

Some processes hold an open file handle on a log file. Deleting the file removes the directory entry but does not release the disk space until the process closes its handle. Instead, truncate the file in place:

# Truncate a specific log file to zero bytes without deleting it
# (the process holding it open will continue writing from byte 0)
> /var/log/apache2/access.log
> /var/log/nginx/access.log
> /var/log/mysql/mysql-slow.log

# Find all log files over 500MB and list them before truncating
find /var/log -type f -size +500M

# Truncate all log files over 500MB in /var/log
find /var/log -type f -size +500M -exec truncate -s 0 {} ;

Docker is one of the most aggressive disk consumers on VPS instances. If Docker is installed, its overlay2 storage can consume tens of gigabytes silently.

# Check total Docker disk usage
docker system df

# Remove stopped containers, dangling images, unused networks, build cache
docker system prune -f

# Also remove unused volumes (more aggressive -- confirm first)
docker system prune --volumes -f

# Remove all unused images, not just dangling ones
docker image prune -a -f

# Check containerd separately (used by some stacks without docker CLI)
du -sh /var/lib/containerd

Use this path if the VNC console login also fails, the system is in a kernel panic, or you need to perform changes that require the filesystem to be unmounted. All paths below use /mnt/recovery as the mount point.

# Mount the production partition read-write
mount /dev/vda1 /mnt/recovery

# Navigate into it
cd /mnt/recovery

# Confirm space
df -h /mnt/recovery

Safe Deletions from Rescue Mode

All paths are relative to /mnt/recovery (your production disk mount). Use full paths so you do not accidentally delete files in the rescue system:

# APT cache
rm -rf /mnt/recovery/var/cache/apt/archives/*.deb

# yum / dnf cache
rm -rf /mnt/recovery/var/cache/yum/*
rm -rf /mnt/recovery/var/cache/dnf/*

# Old compressed logs
find /mnt/recovery/var/log -name '*.gz' -delete
find /mnt/recovery/var/log -name '*.1' -delete
find /mnt/recovery/var/log -name '*.2' -delete

# Tmp files
find /mnt/recovery/tmp -type f -delete
find /mnt/recovery/var/tmp -type f -delete

# Core dumps
find /mnt/recovery/var/crash -type f -delete
find /mnt/recovery -name 'core.*' -type f -delete

# systemd journal
rm -rf /mnt/recovery/var/log/journal/*/

# Docker (if installed) -- removes all images and overlay data
# WARNING: requires Docker to be re-initialized on next boot
du -sh /mnt/recovery/var/lib/docker
rm -rf /mnt/recovery/var/lib/docker/overlay2/*
rm -rf /mnt/recovery/var/lib/docker/image/*

Unmounting and Rebooting

# Confirm you now have free space
df -h /mnt/recovery

# Unmount cleanly
cd /
umount /mnt/recovery

# Reboot back to normal (from RamNode client area)
# Services > My Services > VPS > Boot / Reboot
# Or in rescue console:
reboot

After the reboot, return to the RamNode client area and exit rescue mode if the panel does not do so automatically.

With disk space freed and the server rebooted into its normal OS, SSH should resume accepting connections. If it does not, connect via the VNC console to check the SSH daemon:

# Check if sshd is running
systemctl status sshd

# If it failed, restart it
systemctl restart sshd

# Check for configuration errors from the disk-full incident
sshd -t

# Review sshd logs for clues
journalctl -u sshd --since '30 min ago'

mkdir -p /var/run/sshd && chmod 755 /var/run/sshd && systemctl restart sshd

✓ Safe to Delete

✗ Files You Should NOT Delete

7.1 Monitor Disk Usage with Alerts

# Add to crontab (crontab -e) to alert at 80% usage
DISK_USAGE=$(df / | awk 'NR==2 {print $5}' | tr -d '%')
if [ "$DISK_USAGE" -gt 80 ]; then
  echo "Disk at ${DISK_USAGE}% on $(hostname)" | mail -s "Disk Warning" you@example.com
fi

7.2 Automate Log Rotation

Ensure logrotate is installed and running daily on your VPS:

# Verify logrotate is installed
which logrotate

# Run it manually to test
logrotate -d /etc/logrotate.conf

# Confirm the daily cron job exists
ls -la /etc/cron.daily/logrotate

7.3 Automate Docker Cleanup

# Add to root crontab for weekly cleanup (Sundays at 3am)
0 3 * * 0 docker system prune -f >> /var/log/docker-prune.log 2>&1

7.4 RamNode Upgrade Options

If disk usage is consistently above 75% after cleanup, consider upgrading your RamNode plan. VPS disk upgrades are available without data loss through the RamNode client area under Services → Upgrade/Downgrade. Storage-only upgrades are available if you need more disk without additional CPU or RAM. Resizes complete within minutes and do not require a reinstall.

1. Confirm the problem: run df -h / from any available console.
2. Try VNC console login first via the RamNode client area.
3. If VNC login works, run: apt-get clean, journalctl --vacuum-size=100M, find /var/log -name '*.gz' -delete.
4. Check for runaway logs: find /var/log -type f -size +100M.
5. Check Docker: docker system df, then docker system prune -f.
6. If VNC login fails, boot into Rescue Mode from the RamNode client area.
7. In rescue mode, mount /dev/vda1 to /mnt/recovery and repeat the deletions with full paths.
8. Unmount cleanly (umount /mnt/recovery) and reboot back to normal mode.
9. Verify SSH access is restored; restart sshd if needed.
10. Set up disk usage monitoring to prevent recurrence.