Server Disk Full: Quick Solutions

Immediate assessment and safety steps

When disk space is full, act quickly but safely. First check overall usage with simple commands.

• Run: df -h to see mounted filesystems and percent used.
• Run: df -i to check inode exhaustion — full inodes also block writes.
• Run: du -sh /* 2>/dev/null or du -sh /var /home /usr /tmp 2>/dev/null to get rough folder sizes.

Do not start deleting files blindly. Identify the cause before removing things that services need. If a service logs heavily, restarting it without addressing rotation can cause more problems. If a deleted file still consumes space, find the process holding it with lsof | grep deleted and restart that process or the service.

If the system is critical, alert stakeholders before making changes and consider a quick snapshot or backup if possible.

Identify largest files and directories

Find big directories and files fast so you can target cleanup.

• Use du and sort to spot the top space users:

  du -ahx / | sort -rh | head -n 30
  

• For a more interactive view, install and run ncdu:

  sudo apt install ncdu   # Debian/Ubuntu
  sudo yum install ncdu   # RHEL/CentOS
  sudo ncdu /
  

• Find files over a size threshold:

  find / -xdev -type f -size +100M -exec ls -lh {} ; 2>/dev/null
  

Check common large directories: /var/log, /var/lib/docker, /var/lib/containers, /var/cache, /tmp, /home, and database data directories like /var/lib/mysql or /var/lib/postgresql.

Also inspect hidden files in home directories and large mail spool files (/var/mail).

Clean temporary and cache files

Temporary files and caches are safe to clear most of the time.

• Clear system temp directories:

  sudo rm -rf /tmp/* /var/tmp/*
  

  Be cautious: do not delete files owned by active processes. Prefer cleaning older files:

  find /tmp -type f -mtime +7 -delete
  

• Clean user caches:

  du -sh /home/*/.cache 2>/dev/null
  # Remove large caches if safe:
  rm -rf /home/username/.cache/<large-cache>
  

• Browser and application caches live in users’ home directories; ask users before removing.

• Check container runtimes: Docker and Podman can keep large caches and images (see “Clean package manager and application caches” and Docker section).

Always check file ownership and process use before deletion.

Rotate, truncate, and compress logs

Logs are often the top offender. Rotate, compress, or truncate them.

• Use logrotate to manage logs. Check /etc/logrotate.d/ and configure rotation by size or time:

  /var/log/myapp/*.log {
    daily
    rotate 7
    size 100M
    compress
    missingok
    notifempty
  }
  

• To immediately shrink a huge log without stopping the service:

  sudo truncate -s 0 /var/log/huge.log
  

  Truncate empties the file safely while keeping file handles valid.

• Compress logs older than a certain age:

  find /var/log -name "*.log" -mtime +7 -exec gzip {} ;
  

• For systemd journal logs:

  journalctl --vacuum-size=500M
  journalctl --vacuum-time=7d
  

  Or reduce persistent journal size via /etc/systemd/journald.conf (SystemMaxUse).

Be careful truncating logs that are needed for debugging a current incident; consider archiving first.

Remove unused packages and old kernels

Unused packages and old kernels can accumulate a lot of space.

• Debian/Ubuntu:

  sudo apt autoremove --purge
  sudo apt-get clean
  

  For manual kernel cleanup:

  dpkg --list 'linux-image*' | grep ^ii
  sudo apt remove --purge linux-image-X.Y.Z-...
  sudo update-grub
  

• RHEL/CentOS (yum/dnf):

  sudo dnf remove --oldinstallonly --setopt install_weak_deps=False
  sudo dnf autoremove
  sudo dnf clean all
  

• Arch (pacman):

  sudo pacman -Rns $(pacman -Qtdq)    # orphaned packages
  sudo pacman -Sc                    # clear cache
  

• Snap stores multiple revisions:

  snap list --all | awk '/disabled/{print $1, $3}'
  sudo snap remove <name> --revision=<rev>
  

  Or use a script to keep the last 2 revisions and remove older ones.

Always keep at least one known-good kernel and avoid removing the kernel currently in use.

Clean package manager and application caches

Package caches and app caches are easy wins.

• APT:

  sudo apt-get clean
  sudo apt-get autoclean
  

• Yum/DNF:

  sudo yum clean all
  sudo dnf clean all
  

• Pacman:

  sudo pacman -Scc
  

• Language/runtime caches:

  • Python pip cache: ~/.cache/pip or pip cache purge
  • Node npm cache: npm cache clean --force
  • Ruby gems: gem cleanup

• Docker and container runtimes:

  docker system df
  docker system prune --all --volumes --force
  docker image prune -a
  docker volume prune
  

  For Podman: podman system prune -a

Cleaning caches can break rollback or offline install flows. Only remove caches when acceptable.

Free space by archiving or moving data

If data must be kept, archive or move it off the full filesystem.

• Compress and archive:

  tar -czf /mnt/archive/old-data-2025-11.tar.gz /var/log/old-logs
  

  Compressed archives free space and keep data accessible.

• Move to another filesystem or mount point:

  mkdir /mnt/newdisk/data
  rsync -aXS --progress /var/lib/app/ /mnt/newdisk/data/
  mv /var/lib/app /var/lib/app.bak
  mount --bind /mnt/newdisk/data /var/lib/app
  # After confirming app works, remove the backup
  rm -rf /var/lib/app.bak
  

  Use rsync to preserve permissions and reduce downtime.

• Offload to cloud object storage (S3) or network storage (NFS). Use tools like rclone for S3.

Always validate data integrity after moving. Keep backups until you are certain.

Expand storage: resize volumes or attach disks

When cleanup isn’t enough, increase available storage.

• Cloud VMs:

  • Attach another disk from the cloud console and mount it.
  • Or increase the disk size, then grow the partition and filesystem (growpart, resize2fs, xfs_growfs).

• LVM volumes:

  pvcreate /dev/sdb
  vgextend vg0 /dev/sdb
  lvextend -L +50G /dev/vg0/root
  resize2fs /dev/vg0/root   # or xfs_growfs /mountpoint for XFS
  

• Partition resizing (careful):

  • Use parted/gparted for offline resizing.
  • Always have backups before changing partitions.

• For XFS filesystems, use xfs_growfs after extending the block device; for ext4, use resize2fs.

Expanding storage is the long-term fix when data growth is expected.

Short-term fixes: bind mounts and temporary mounts

Bind mounts let you move heavy directories to other disks without changing apps.

• Steps to move a directory with minimal downtime:

  # Prepare new disk and mount it at /mnt/newdisk
  rsync -aXS /var/lib/mysql/ /mnt/newdisk/mysql/
  mv /var/lib/mysql /var/lib/mysql.bak
  mkdir /var/lib/mysql
  mount --bind /mnt/newdisk/mysql /var/lib/mysql
  # Add to /etc/fstab:
  /mnt/newdisk/mysql /var/lib/mysql none bind 0 0
  

  After testing, remove the backup.

• Temporary mounts:

  • Mount a temporary disk to /tmp or /var/log to relieve pressure quickly.
  • This is a stopgap; document the change and plan a permanent move.

Make sure file permissions and SELinux contexts are preserved when moving directories.

Implement monitoring, alerts, and auto-remediation

Prevent recurrence by watching disk usage and acting early.

• Monitoring tools:

  • Prometheus node_exporter, Grafana dashboards.
  • Cloud monitoring: AWS CloudWatch, Azure Monitor, Google Cloud Monitoring.
  • Traditional: Nagios, Zabbix.

• Alerts to set:

  • Filesystem percent used (e.g., warn at 70%, critical at 90%).
  • Rapid growth detection on log directories.
  • Inode usage alerts.

• Auto-remediation examples:

  • A script that prunes Docker images when disk > 80% (run from cron or an orchestration tool).
  • systemd timers or cronjobs to rotate old logs, vacuum journal, and clean caches.
  • Automated retention policies for CI artifacts and backups.

Auto-remediation should be conservative and logged. Always alert humans before destructive cleanup if possible.

Policies and best practices to prevent recurrence

Implement rules that keep disk usage healthy over time.

• Define retention policies for logs, artifacts, and backups. Automate enforcement.
• Centralize logs using a log aggregation system (ELK/EFK, Splunk, cloud logging). This reduces local disk use.
• Use quotas for users and services to limit runaway growth.
• Limit container image retention and use slim base images.
• Enforce package cleanup in build and CI pipelines.
• Regularly audit filesystem usage and review large file owners.
• Test patching and kernel updates in staging; automate old-kernel removal policy.

Clear policies make it easier to scale without surprise outages.

Post-resolution checklist and documentation

After you free space, do these checks and document everything.

• Confirm free space:

  df -h
  df -i
  

• Check services are healthy and logs are being written properly.
• Verify no processes are holding deleted files (lsof | grep deleted).
• Ensure backups are running and integrity checks pass.
• Record root cause: what grew, why, and how it was fixed.
• Document the steps taken, commands run, and any configuration changes.
• Create follow-up tasks: adjust logrotate, add monitoring alerts, increase disk size if needed.
• Communicate with stakeholders what changed and any follow-up impacts.

Good documentation speeds recovery next time and helps avoid surprises.

This guide gives practical, safe steps to reclaim disk space, handle short-term emergencies, and put long-term fixes in place. Use the commands carefully, test changes when possible, and keep backups and logs until you confirm the system is stable.