Add comprehensive backup system documentation
Document complete backup strategy covering: - Setup and initialization procedures - What gets backed up (and what doesn't) - Multiple restore scenarios (full reinstall, rollback, specific volumes) - Retention policy explanation - Troubleshooting and security notes
This commit is contained in:
233
docs/system-backups.md
Normal file
233
docs/system-backups.md
Normal file
@@ -0,0 +1,233 @@
|
||||
# System Backups
|
||||
|
||||
Complete backup strategy for the homelab using Restic for encrypted, deduplicated backups to NAS.
|
||||
|
||||
## Overview
|
||||
|
||||
**Backup Tool**: Restic (modern, encrypted, deduplicated)
|
||||
**Backup Destination**: NAS at `/mnt/nas/backups/homelab-restic`
|
||||
**Backup Frequency**: Daily (can be automated with systemd timer)
|
||||
**Encryption**: AES-256, password stored in `/home/hoborg/creds/restic-password.txt`
|
||||
|
||||
## What Gets Backed Up
|
||||
|
||||
### Full System Backup Includes:
|
||||
1. **Package Lists**
|
||||
- All explicitly installed packages (pacman)
|
||||
- All installed packages (including dependencies)
|
||||
- AUR/foreign packages list
|
||||
- Allows exact system recreation after reinstall
|
||||
|
||||
2. **System Configuration** (`/etc/`)
|
||||
- Nginx configuration
|
||||
- Systemd services
|
||||
- Network configuration
|
||||
- All system-wide configs
|
||||
- Excludes: shadow/gshadow password files
|
||||
|
||||
3. **Docker Volumes**
|
||||
- Gitea (Git repositories and database)
|
||||
- Jellyfin (media library configuration)
|
||||
- Nextcloud (cloud storage data and database)
|
||||
- Portainer (container management data)
|
||||
|
||||
### What's NOT Backed Up (Intentionally):
|
||||
- **User data in `/home`**: Separate partition, safe from system reinstalls
|
||||
- **Media files**: Already stored on NAS (Music, Videos, Pictures)
|
||||
- **User documents**: Synced via Nextcloud/Copyparty or on separate partition
|
||||
|
||||
## Setup
|
||||
|
||||
### Initial Setup (One-Time)
|
||||
|
||||
1. **Initialize the backup repository:**
|
||||
```bash
|
||||
sudo /home/hoborg/homelab/scripts/backup-init.sh
|
||||
```
|
||||
|
||||
This will:
|
||||
- Install Restic if needed
|
||||
- Create backup directory on NAS (`/mnt/nas/backups/homelab-restic`)
|
||||
- Generate encryption password (saved to `/home/hoborg/creds/restic-password.txt`)
|
||||
- Initialize the Restic repository
|
||||
|
||||
⚠️ **IMPORTANT**: Back up the password file! Without it, backups cannot be restored.
|
||||
|
||||
### Running Backups
|
||||
|
||||
**Manual backup:**
|
||||
```bash
|
||||
sudo /home/hoborg/homelab/scripts/backup-system-full.sh
|
||||
```
|
||||
|
||||
**Automated backups** (optional - setup systemd timer later):
|
||||
- Create systemd service and timer
|
||||
- Schedule daily backups (e.g., 3 AM)
|
||||
- Logs written to `/var/log/homelab-backup.log`
|
||||
|
||||
## Backup Scripts
|
||||
|
||||
### `backup-init.sh`
|
||||
Initializes the Restic repository. Run once during setup.
|
||||
|
||||
Location: `/home/hoborg/homelab/scripts/backup-init.sh`
|
||||
|
||||
### `backup-system-full.sh`
|
||||
Performs full system backup including packages, configs, and Docker volumes.
|
||||
|
||||
Location: `/home/hoborg/homelab/scripts/backup-system-full.sh`
|
||||
|
||||
What it does:
|
||||
1. Exports package lists (pacman explicit, all packages, AUR packages)
|
||||
2. Exports Docker volumes to compressed tarballs
|
||||
3. Backs up entire `/etc/` directory
|
||||
4. Prunes old backups (keeps: 7 daily, 4 weekly, 6 monthly, 2 yearly)
|
||||
5. Shows backup statistics
|
||||
|
||||
## Restore Procedures
|
||||
|
||||
### List Available Backups
|
||||
|
||||
```bash
|
||||
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
|
||||
restic -r /mnt/nas/backups/homelab-restic snapshots
|
||||
```
|
||||
|
||||
### Restore After System Reinstall
|
||||
|
||||
1. **Restore backup to temporary location:**
|
||||
```bash
|
||||
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
|
||||
restic -r /mnt/nas/backups/homelab-restic restore latest --target /tmp/restore
|
||||
```
|
||||
|
||||
2. **Reinstall packages:**
|
||||
```bash
|
||||
# Reinstall all explicitly installed packages
|
||||
sudo pacman -S --needed $(cat /tmp/restore/tmp/system-backup-*/pacman-explicit.txt)
|
||||
|
||||
# Reinstall AUR packages
|
||||
yay -S --needed $(cat /tmp/restore/tmp/system-backup-*/aur-packages.txt)
|
||||
```
|
||||
|
||||
3. **Restore system configs:**
|
||||
```bash
|
||||
sudo cp -r /tmp/restore/etc/* /etc/
|
||||
sudo systemctl daemon-reload
|
||||
```
|
||||
|
||||
4. **Restore Docker volumes:**
|
||||
```bash
|
||||
# Example: Restore Gitea
|
||||
docker-compose -f /opt/docker/gitea/docker-compose.yml down
|
||||
|
||||
sudo tar xzf /tmp/restore/tmp/docker-backup-*/gitea-data.tar.gz -C /tmp/gitea-restore
|
||||
|
||||
docker run --rm -v gitea_gitea:/data -v /tmp/gitea-restore:/backup alpine \
|
||||
sh -c "rm -rf /data/* && cp -a /backup/* /data/"
|
||||
|
||||
docker-compose -f /opt/docker/gitea/docker-compose.yml up -d
|
||||
```
|
||||
|
||||
### Rollback After Broken Update
|
||||
|
||||
If a system update breaks something:
|
||||
|
||||
1. **Restore specific config:**
|
||||
```bash
|
||||
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
|
||||
|
||||
# Restore nginx config
|
||||
restic -r /mnt/nas/backups/homelab-restic restore latest \
|
||||
--target /tmp/restore --include /etc/nginx/
|
||||
|
||||
sudo cp -r /tmp/restore/etc/nginx/* /etc/nginx/
|
||||
sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
2. **Downgrade packages:**
|
||||
```bash
|
||||
# Check which packages were updated
|
||||
grep -A5 "upgraded" /var/log/pacman.log | tail -20
|
||||
|
||||
# Downgrade specific package to cached version
|
||||
sudo pacman -U /var/cache/pacman/pkg/package-old-version.pkg.tar.zst
|
||||
```
|
||||
|
||||
### Restore Specific Docker Volume
|
||||
|
||||
```bash
|
||||
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
|
||||
|
||||
# List files in backup
|
||||
restic -r /mnt/nas/backups/homelab-restic ls latest | grep docker-backup
|
||||
|
||||
# Restore specific volume tarball
|
||||
restic -r /mnt/nas/backups/homelab-restic restore latest \
|
||||
--target /tmp/restore \
|
||||
--include /tmp/docker-backup-*/nextcloud-data.tar.gz
|
||||
|
||||
# Extract and restore to Docker volume
|
||||
mkdir /tmp/nextcloud-data
|
||||
tar xzf /tmp/restore/tmp/docker-backup-*/nextcloud-data.tar.gz -C /tmp/nextcloud-data
|
||||
|
||||
docker run --rm -v nextcloud_nextcloud_data:/data -v /tmp/nextcloud-data:/backup alpine \
|
||||
sh -c "rm -rf /data/* && cp -a /backup/* /data/"
|
||||
```
|
||||
|
||||
## Backup Retention Policy
|
||||
|
||||
The backup script automatically prunes old backups:
|
||||
- **Daily**: Keep last 7 days
|
||||
- **Weekly**: Keep last 4 weeks
|
||||
- **Monthly**: Keep last 6 months
|
||||
- **Yearly**: Keep last 2 years
|
||||
|
||||
This balances storage space with recovery options.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Repository not initialized" Error
|
||||
|
||||
Run the initialization script:
|
||||
```bash
|
||||
sudo /home/hoborg/homelab/scripts/backup-init.sh
|
||||
```
|
||||
|
||||
### "Password incorrect" Error
|
||||
|
||||
Check that password file exists and is readable:
|
||||
```bash
|
||||
ls -l /home/hoborg/creds/restic-password.txt
|
||||
```
|
||||
|
||||
### Check Backup Size
|
||||
|
||||
```bash
|
||||
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
|
||||
restic -r /mnt/nas/backups/homelab-restic stats --mode restore-size
|
||||
```
|
||||
|
||||
### Verify Backup Integrity
|
||||
|
||||
```bash
|
||||
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
|
||||
restic -r /mnt/nas/backups/homelab-restic check
|
||||
```
|
||||
|
||||
## Security Notes
|
||||
|
||||
- Backups are encrypted with AES-256
|
||||
- Password file has `600` permissions (owner read/write only)
|
||||
- Shadow/gshadow files are excluded from backups
|
||||
- NAS should have access restrictions
|
||||
- Consider offsite backup copy of password file
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [ ] Automated systemd timer setup
|
||||
- [ ] Pre-backup notifications
|
||||
- [ ] Post-backup success/failure notifications
|
||||
- [ ] Backup verification script
|
||||
- [ ] Web dashboard for backup status
|
||||
- [ ] Offsite backup replication (optional)
|
||||
Reference in New Issue
Block a user