homelab/docs/system-backups.md

# System Backups

Complete backup strategy for the homelab using Restic for encrypted, deduplicated backups to NAS.

## Overview

**Backup Tool**: Restic (modern, encrypted, deduplicated)
**Backup Destination**: NAS at `/mnt/nas/backups/homelab-restic`
**Backup Frequency**: Daily (can be automated with systemd timer)
**Encryption**: AES-256, password stored in `/home/hoborg/creds/restic-password.txt`

## What Gets Backed Up

### Full System Backup Includes:
1. **Package Lists**
   - All explicitly installed packages (pacman)
   - All installed packages (including dependencies)
   - AUR/foreign packages list
   - Allows exact system recreation after reinstall

2. **System Configuration** (`/etc/`)
   - Nginx configuration
   - Systemd services
   - Network configuration
   - All system-wide configs
   - Excludes: shadow/gshadow password files

3. **Docker Volumes**
   - Gitea (Git repositories and database)
   - Jellyfin (media library configuration)
   - Nextcloud (cloud storage data and database)
   - Portainer (container management data)

### What's NOT Backed Up (Intentionally):
- **User data in `/home`**: Separate partition, safe from system reinstalls
- **Media files**: Already stored on NAS (Music, Videos, Pictures)
- **User documents**: Synced via Nextcloud/Copyparty or on separate partition

## Setup

### Initial Setup (One-Time)

1. **Initialize the backup repository:**
   ```bash
   sudo /home/hoborg/homelab/scripts/backup-init.sh
   ```

   This will:
   - Install Restic if needed
   - Create backup directory on NAS (`/mnt/nas/backups/homelab-restic`)
   - Generate encryption password (saved to `/home/hoborg/creds/restic-password.txt`)
   - Initialize the Restic repository

   ⚠️ **IMPORTANT**: Back up the password file! Without it, backups cannot be restored.

### Running Backups

**Manual backup:**
```bash
sudo /home/hoborg/homelab/scripts/backup-system-full.sh
```

**Automated backups** (optional - setup systemd timer later):
- Create systemd service and timer
- Schedule daily backups (e.g., 3 AM)
- Logs written to `/var/log/homelab-backup.log`

## Backup Scripts

### `backup-init.sh`
Initializes the Restic repository. Run once during setup.

Location: `/home/hoborg/homelab/scripts/backup-init.sh`

### `backup-system-full.sh`
Performs full system backup including packages, configs, and Docker volumes.

Location: `/home/hoborg/homelab/scripts/backup-system-full.sh`

What it does:
1. Exports package lists (pacman explicit, all packages, AUR packages)
2. Exports Docker volumes to compressed tarballs
3. Backs up entire `/etc/` directory
4. Prunes old backups (keeps: 7 daily, 4 weekly, 6 monthly, 2 yearly)
5. Shows backup statistics

## Restore Procedures

### List Available Backups

```bash
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
restic -r /mnt/nas/backups/homelab-restic snapshots
```

### Restore After System Reinstall

1. **Restore backup to temporary location:**
   ```bash
   export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
   restic -r /mnt/nas/backups/homelab-restic restore latest --target /tmp/restore
   ```

2. **Reinstall packages:**
   ```bash
   # Reinstall all explicitly installed packages
   sudo pacman -S --needed $(cat /tmp/restore/tmp/system-backup-*/pacman-explicit.txt)

   # Reinstall AUR packages
   yay -S --needed $(cat /tmp/restore/tmp/system-backup-*/aur-packages.txt)
   ```

3. **Restore system configs:**
   ```bash
   sudo cp -r /tmp/restore/etc/* /etc/
   sudo systemctl daemon-reload
   ```

4. **Restore Docker volumes:**
   ```bash
   # Example: Restore Gitea
   docker-compose -f /opt/docker/gitea/docker-compose.yml down

   sudo tar xzf /tmp/restore/tmp/docker-backup-*/gitea-data.tar.gz -C /tmp/gitea-restore

   docker run --rm -v gitea_gitea:/data -v /tmp/gitea-restore:/backup alpine \
     sh -c "rm -rf /data/* && cp -a /backup/* /data/"

   docker-compose -f /opt/docker/gitea/docker-compose.yml up -d
   ```

### Rollback After Broken Update

If a system update breaks something:

1. **Restore specific config:**
   ```bash
   export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"

   # Restore nginx config
   restic -r /mnt/nas/backups/homelab-restic restore latest \
     --target /tmp/restore --include /etc/nginx/

   sudo cp -r /tmp/restore/etc/nginx/* /etc/nginx/
   sudo systemctl reload nginx
   ```

2. **Downgrade packages:**
   ```bash
   # Check which packages were updated
   grep -A5 "upgraded" /var/log/pacman.log | tail -20

   # Downgrade specific package to cached version
   sudo pacman -U /var/cache/pacman/pkg/package-old-version.pkg.tar.zst
   ```

### Restore Specific Docker Volume

```bash
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"

# List files in backup
restic -r /mnt/nas/backups/homelab-restic ls latest | grep docker-backup

# Restore specific volume tarball
restic -r /mnt/nas/backups/homelab-restic restore latest \
  --target /tmp/restore \
  --include /tmp/docker-backup-*/nextcloud-data.tar.gz

# Extract and restore to Docker volume
mkdir /tmp/nextcloud-data
tar xzf /tmp/restore/tmp/docker-backup-*/nextcloud-data.tar.gz -C /tmp/nextcloud-data

docker run --rm -v nextcloud_nextcloud_data:/data -v /tmp/nextcloud-data:/backup alpine \
  sh -c "rm -rf /data/* && cp -a /backup/* /data/"
```

## Backup Retention Policy

The backup script automatically prunes old backups:
- **Daily**: Keep last 7 days
- **Weekly**: Keep last 4 weeks
- **Monthly**: Keep last 6 months
- **Yearly**: Keep last 2 years

This balances storage space with recovery options.

## Troubleshooting

### "Repository not initialized" Error

Run the initialization script:
```bash
sudo /home/hoborg/homelab/scripts/backup-init.sh
```

### "Password incorrect" Error

Check that password file exists and is readable:
```bash
ls -l /home/hoborg/creds/restic-password.txt
```

### Check Backup Size

```bash
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
restic -r /mnt/nas/backups/homelab-restic stats --mode restore-size
```

### Verify Backup Integrity

```bash
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
restic -r /mnt/nas/backups/homelab-restic check
```

## Security Notes

- Backups are encrypted with AES-256
- Password file has `600` permissions (owner read/write only)
- Shadow/gshadow files are excluded from backups
- NAS should have access restrictions
- Consider offsite backup copy of password file

## Future Enhancements

- [ ] Automated systemd timer setup
- [ ] Pre-backup notifications
- [ ] Post-backup success/failure notifications
- [ ] Backup verification script
- [ ] Web dashboard for backup status
- [ ] Offsite backup replication (optional)