homelab/docs/system-backups.md

System Backups

Complete backup strategy for the homelab using Restic for encrypted, deduplicated backups to NAS.

Overview

Backup Tool: Restic (modern, encrypted, deduplicated) Backup Destination: NAS at /mnt/nas/backups/homelab-restic Backup Frequency: Daily (can be automated with systemd timer) Encryption: AES-256, password stored in /home/hoborg/creds/restic-password.txt

What Gets Backed Up

Full System Backup Includes:

1. Package Lists

   • All explicitly installed packages (pacman)
   • All installed packages (including dependencies)
   • AUR/foreign packages list
   • Allows exact system recreation after reinstall

2. System Configuration (/etc/)

   • Nginx configuration
   • Systemd services
   • Network configuration
   • All system-wide configs
   • Excludes: shadow/gshadow password files

3. Docker Volumes

   • Gitea (Git repositories and database)
   • Jellyfin (media library configuration)
   • Nextcloud (cloud storage data and database)
   • Portainer (container management data)

What's NOT Backed Up (Intentionally):

• User data in /home: Separate partition, safe from system reinstalls
• Media files: Already stored on NAS (Music, Videos, Pictures)
• User documents: Synced via Nextcloud/Copyparty or on separate partition

Setup

Initial Setup (One-Time)

1. Initialize the backup repository:

   sudo /home/hoborg/homelab/scripts/backup-init.sh
   

   This will:

   • Install Restic if needed
   • Create backup directory on NAS (/mnt/nas/backups/homelab-restic)
   • Generate encryption password (saved to /home/hoborg/creds/restic-password.txt)
   • Initialize the Restic repository

   ⚠️ IMPORTANT: Back up the password file! Without it, backups cannot be restored.

Running Backups

Manual backup:

sudo /home/hoborg/homelab/scripts/backup-system-full.sh

Automated backups (optional - setup systemd timer later):

• Create systemd service and timer
• Schedule daily backups (e.g., 3 AM)
• Logs written to /var/log/homelab-backup.log

Backup Scripts

backup-init.sh

Initializes the Restic repository. Run once during setup.

Location: /home/hoborg/homelab/scripts/backup-init.sh

backup-system-full.sh

Performs full system backup including packages, configs, and Docker volumes.

Location: /home/hoborg/homelab/scripts/backup-system-full.sh

What it does:

1. Exports package lists (pacman explicit, all packages, AUR packages)
2. Exports Docker volumes to compressed tarballs
3. Backs up entire /etc/ directory
4. Prunes old backups (keeps: 7 daily, 4 weekly, 6 monthly, 2 yearly)
5. Shows backup statistics

Restore Procedures

List Available Backups

export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
restic -r /mnt/nas/backups/homelab-restic snapshots

Restore After System Reinstall

1. Restore backup to temporary location:

   export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
   restic -r /mnt/nas/backups/homelab-restic restore latest --target /tmp/restore
   

2. Reinstall packages:

   # Reinstall all explicitly installed packages
   sudo pacman -S --needed $(cat /tmp/restore/tmp/system-backup-*/pacman-explicit.txt)
   
   # Reinstall AUR packages
   yay -S --needed $(cat /tmp/restore/tmp/system-backup-*/aur-packages.txt)
   

3. Restore system configs:

   sudo cp -r /tmp/restore/etc/* /etc/
   sudo systemctl daemon-reload
   

4. Restore Docker volumes:

   # Example: Restore Gitea
   docker-compose -f /opt/docker/gitea/docker-compose.yml down
   
   sudo tar xzf /tmp/restore/tmp/docker-backup-*/gitea-data.tar.gz -C /tmp/gitea-restore
   
   docker run --rm -v gitea_gitea:/data -v /tmp/gitea-restore:/backup alpine \
     sh -c "rm -rf /data/* && cp -a /backup/* /data/"
   
   docker-compose -f /opt/docker/gitea/docker-compose.yml up -d
   

Rollback After Broken Update

If a system update breaks something:

1. Restore specific config:

   export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
   
   # Restore nginx config
   restic -r /mnt/nas/backups/homelab-restic restore latest \
     --target /tmp/restore --include /etc/nginx/
   
   sudo cp -r /tmp/restore/etc/nginx/* /etc/nginx/
   sudo systemctl reload nginx
   

2. Downgrade packages:

   # Check which packages were updated
   grep -A5 "upgraded" /var/log/pacman.log | tail -20
   
   # Downgrade specific package to cached version
   sudo pacman -U /var/cache/pacman/pkg/package-old-version.pkg.tar.zst
   

Restore Specific Docker Volume

export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"

# List files in backup
restic -r /mnt/nas/backups/homelab-restic ls latest | grep docker-backup

# Restore specific volume tarball
restic -r /mnt/nas/backups/homelab-restic restore latest \
  --target /tmp/restore \
  --include /tmp/docker-backup-*/nextcloud-data.tar.gz

# Extract and restore to Docker volume
mkdir /tmp/nextcloud-data
tar xzf /tmp/restore/tmp/docker-backup-*/nextcloud-data.tar.gz -C /tmp/nextcloud-data

docker run --rm -v nextcloud_nextcloud_data:/data -v /tmp/nextcloud-data:/backup alpine \
  sh -c "rm -rf /data/* && cp -a /backup/* /data/"

Backup Retention Policy

The backup script automatically prunes old backups:

• Daily: Keep last 7 days
• Weekly: Keep last 4 weeks
• Monthly: Keep last 6 months
• Yearly: Keep last 2 years

This balances storage space with recovery options.

Troubleshooting

"Repository not initialized" Error

Run the initialization script:

sudo /home/hoborg/homelab/scripts/backup-init.sh

"Password incorrect" Error

Check that password file exists and is readable:

ls -l /home/hoborg/creds/restic-password.txt

Check Backup Size

export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
restic -r /mnt/nas/backups/homelab-restic stats --mode restore-size

Verify Backup Integrity

export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
restic -r /mnt/nas/backups/homelab-restic check

Security Notes

• Backups are encrypted with AES-256
• Password file has 600 permissions (owner read/write only)
• Shadow/gshadow files are excluded from backups
• NAS should have access restrictions
• Consider offsite backup copy of password file

Future Enhancements

• Automated systemd timer setup
• Pre-backup notifications
• Post-backup success/failure notifications
• Backup verification script
• Web dashboard for backup status
• Offsite backup replication (optional)