From ab4a4cfc9cbcaf11d6cd445272720d1fbcc04c14 Mon Sep 17 00:00:00 2001 From: Arpad Krejczinger Date: Sat, 11 Oct 2025 18:24:58 +0200 Subject: [PATCH] Add comprehensive backup system documentation Document complete backup strategy covering: - Setup and initialization procedures - What gets backed up (and what doesn't) - Multiple restore scenarios (full reinstall, rollback, specific volumes) - Retention policy explanation - Troubleshooting and security notes --- docs/system-backups.md | 233 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 233 insertions(+) create mode 100644 docs/system-backups.md diff --git a/docs/system-backups.md b/docs/system-backups.md new file mode 100644 index 0000000..19acde2 --- /dev/null +++ b/docs/system-backups.md @@ -0,0 +1,233 @@ +# System Backups + +Complete backup strategy for the homelab using Restic for encrypted, deduplicated backups to NAS. + +## Overview + +**Backup Tool**: Restic (modern, encrypted, deduplicated) +**Backup Destination**: NAS at `/mnt/nas/backups/homelab-restic` +**Backup Frequency**: Daily (can be automated with systemd timer) +**Encryption**: AES-256, password stored in `/home/hoborg/creds/restic-password.txt` + +## What Gets Backed Up + +### Full System Backup Includes: +1. **Package Lists** + - All explicitly installed packages (pacman) + - All installed packages (including dependencies) + - AUR/foreign packages list + - Allows exact system recreation after reinstall + +2. **System Configuration** (`/etc/`) + - Nginx configuration + - Systemd services + - Network configuration + - All system-wide configs + - Excludes: shadow/gshadow password files + +3. **Docker Volumes** + - Gitea (Git repositories and database) + - Jellyfin (media library configuration) + - Nextcloud (cloud storage data and database) + - Portainer (container management data) + +### What's NOT Backed Up (Intentionally): +- **User data in `/home`**: Separate partition, safe from system reinstalls +- **Media files**: Already stored on NAS (Music, Videos, Pictures) +- **User documents**: Synced via Nextcloud/Copyparty or on separate partition + +## Setup + +### Initial Setup (One-Time) + +1. **Initialize the backup repository:** + ```bash + sudo /home/hoborg/homelab/scripts/backup-init.sh + ``` + + This will: + - Install Restic if needed + - Create backup directory on NAS (`/mnt/nas/backups/homelab-restic`) + - Generate encryption password (saved to `/home/hoborg/creds/restic-password.txt`) + - Initialize the Restic repository + + ⚠️ **IMPORTANT**: Back up the password file! Without it, backups cannot be restored. + +### Running Backups + +**Manual backup:** +```bash +sudo /home/hoborg/homelab/scripts/backup-system-full.sh +``` + +**Automated backups** (optional - setup systemd timer later): +- Create systemd service and timer +- Schedule daily backups (e.g., 3 AM) +- Logs written to `/var/log/homelab-backup.log` + +## Backup Scripts + +### `backup-init.sh` +Initializes the Restic repository. Run once during setup. + +Location: `/home/hoborg/homelab/scripts/backup-init.sh` + +### `backup-system-full.sh` +Performs full system backup including packages, configs, and Docker volumes. + +Location: `/home/hoborg/homelab/scripts/backup-system-full.sh` + +What it does: +1. Exports package lists (pacman explicit, all packages, AUR packages) +2. Exports Docker volumes to compressed tarballs +3. Backs up entire `/etc/` directory +4. Prunes old backups (keeps: 7 daily, 4 weekly, 6 monthly, 2 yearly) +5. Shows backup statistics + +## Restore Procedures + +### List Available Backups + +```bash +export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt" +restic -r /mnt/nas/backups/homelab-restic snapshots +``` + +### Restore After System Reinstall + +1. **Restore backup to temporary location:** + ```bash + export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt" + restic -r /mnt/nas/backups/homelab-restic restore latest --target /tmp/restore + ``` + +2. **Reinstall packages:** + ```bash + # Reinstall all explicitly installed packages + sudo pacman -S --needed $(cat /tmp/restore/tmp/system-backup-*/pacman-explicit.txt) + + # Reinstall AUR packages + yay -S --needed $(cat /tmp/restore/tmp/system-backup-*/aur-packages.txt) + ``` + +3. **Restore system configs:** + ```bash + sudo cp -r /tmp/restore/etc/* /etc/ + sudo systemctl daemon-reload + ``` + +4. **Restore Docker volumes:** + ```bash + # Example: Restore Gitea + docker-compose -f /opt/docker/gitea/docker-compose.yml down + + sudo tar xzf /tmp/restore/tmp/docker-backup-*/gitea-data.tar.gz -C /tmp/gitea-restore + + docker run --rm -v gitea_gitea:/data -v /tmp/gitea-restore:/backup alpine \ + sh -c "rm -rf /data/* && cp -a /backup/* /data/" + + docker-compose -f /opt/docker/gitea/docker-compose.yml up -d + ``` + +### Rollback After Broken Update + +If a system update breaks something: + +1. **Restore specific config:** + ```bash + export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt" + + # Restore nginx config + restic -r /mnt/nas/backups/homelab-restic restore latest \ + --target /tmp/restore --include /etc/nginx/ + + sudo cp -r /tmp/restore/etc/nginx/* /etc/nginx/ + sudo systemctl reload nginx + ``` + +2. **Downgrade packages:** + ```bash + # Check which packages were updated + grep -A5 "upgraded" /var/log/pacman.log | tail -20 + + # Downgrade specific package to cached version + sudo pacman -U /var/cache/pacman/pkg/package-old-version.pkg.tar.zst + ``` + +### Restore Specific Docker Volume + +```bash +export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt" + +# List files in backup +restic -r /mnt/nas/backups/homelab-restic ls latest | grep docker-backup + +# Restore specific volume tarball +restic -r /mnt/nas/backups/homelab-restic restore latest \ + --target /tmp/restore \ + --include /tmp/docker-backup-*/nextcloud-data.tar.gz + +# Extract and restore to Docker volume +mkdir /tmp/nextcloud-data +tar xzf /tmp/restore/tmp/docker-backup-*/nextcloud-data.tar.gz -C /tmp/nextcloud-data + +docker run --rm -v nextcloud_nextcloud_data:/data -v /tmp/nextcloud-data:/backup alpine \ + sh -c "rm -rf /data/* && cp -a /backup/* /data/" +``` + +## Backup Retention Policy + +The backup script automatically prunes old backups: +- **Daily**: Keep last 7 days +- **Weekly**: Keep last 4 weeks +- **Monthly**: Keep last 6 months +- **Yearly**: Keep last 2 years + +This balances storage space with recovery options. + +## Troubleshooting + +### "Repository not initialized" Error + +Run the initialization script: +```bash +sudo /home/hoborg/homelab/scripts/backup-init.sh +``` + +### "Password incorrect" Error + +Check that password file exists and is readable: +```bash +ls -l /home/hoborg/creds/restic-password.txt +``` + +### Check Backup Size + +```bash +export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt" +restic -r /mnt/nas/backups/homelab-restic stats --mode restore-size +``` + +### Verify Backup Integrity + +```bash +export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt" +restic -r /mnt/nas/backups/homelab-restic check +``` + +## Security Notes + +- Backups are encrypted with AES-256 +- Password file has `600` permissions (owner read/write only) +- Shadow/gshadow files are excluded from backups +- NAS should have access restrictions +- Consider offsite backup copy of password file + +## Future Enhancements + +- [ ] Automated systemd timer setup +- [ ] Pre-backup notifications +- [ ] Post-backup success/failure notifications +- [ ] Backup verification script +- [ ] Web dashboard for backup status +- [ ] Offsite backup replication (optional)