Add comprehensive backup system documentation
Document complete backup strategy covering: - Setup and initialization procedures - What gets backed up (and what doesn't) - Multiple restore scenarios (full reinstall, rollback, specific volumes) - Retention policy explanation - Troubleshooting and security notes
This commit is contained in:
233
docs/system-backups.md
Normal file
233
docs/system-backups.md
Normal file
@@ -0,0 +1,233 @@
|
|||||||
|
# System Backups
|
||||||
|
|
||||||
|
Complete backup strategy for the homelab using Restic for encrypted, deduplicated backups to NAS.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
**Backup Tool**: Restic (modern, encrypted, deduplicated)
|
||||||
|
**Backup Destination**: NAS at `/mnt/nas/backups/homelab-restic`
|
||||||
|
**Backup Frequency**: Daily (can be automated with systemd timer)
|
||||||
|
**Encryption**: AES-256, password stored in `/home/hoborg/creds/restic-password.txt`
|
||||||
|
|
||||||
|
## What Gets Backed Up
|
||||||
|
|
||||||
|
### Full System Backup Includes:
|
||||||
|
1. **Package Lists**
|
||||||
|
- All explicitly installed packages (pacman)
|
||||||
|
- All installed packages (including dependencies)
|
||||||
|
- AUR/foreign packages list
|
||||||
|
- Allows exact system recreation after reinstall
|
||||||
|
|
||||||
|
2. **System Configuration** (`/etc/`)
|
||||||
|
- Nginx configuration
|
||||||
|
- Systemd services
|
||||||
|
- Network configuration
|
||||||
|
- All system-wide configs
|
||||||
|
- Excludes: shadow/gshadow password files
|
||||||
|
|
||||||
|
3. **Docker Volumes**
|
||||||
|
- Gitea (Git repositories and database)
|
||||||
|
- Jellyfin (media library configuration)
|
||||||
|
- Nextcloud (cloud storage data and database)
|
||||||
|
- Portainer (container management data)
|
||||||
|
|
||||||
|
### What's NOT Backed Up (Intentionally):
|
||||||
|
- **User data in `/home`**: Separate partition, safe from system reinstalls
|
||||||
|
- **Media files**: Already stored on NAS (Music, Videos, Pictures)
|
||||||
|
- **User documents**: Synced via Nextcloud/Copyparty or on separate partition
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
|
||||||
|
### Initial Setup (One-Time)
|
||||||
|
|
||||||
|
1. **Initialize the backup repository:**
|
||||||
|
```bash
|
||||||
|
sudo /home/hoborg/homelab/scripts/backup-init.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
This will:
|
||||||
|
- Install Restic if needed
|
||||||
|
- Create backup directory on NAS (`/mnt/nas/backups/homelab-restic`)
|
||||||
|
- Generate encryption password (saved to `/home/hoborg/creds/restic-password.txt`)
|
||||||
|
- Initialize the Restic repository
|
||||||
|
|
||||||
|
⚠️ **IMPORTANT**: Back up the password file! Without it, backups cannot be restored.
|
||||||
|
|
||||||
|
### Running Backups
|
||||||
|
|
||||||
|
**Manual backup:**
|
||||||
|
```bash
|
||||||
|
sudo /home/hoborg/homelab/scripts/backup-system-full.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
**Automated backups** (optional - setup systemd timer later):
|
||||||
|
- Create systemd service and timer
|
||||||
|
- Schedule daily backups (e.g., 3 AM)
|
||||||
|
- Logs written to `/var/log/homelab-backup.log`
|
||||||
|
|
||||||
|
## Backup Scripts
|
||||||
|
|
||||||
|
### `backup-init.sh`
|
||||||
|
Initializes the Restic repository. Run once during setup.
|
||||||
|
|
||||||
|
Location: `/home/hoborg/homelab/scripts/backup-init.sh`
|
||||||
|
|
||||||
|
### `backup-system-full.sh`
|
||||||
|
Performs full system backup including packages, configs, and Docker volumes.
|
||||||
|
|
||||||
|
Location: `/home/hoborg/homelab/scripts/backup-system-full.sh`
|
||||||
|
|
||||||
|
What it does:
|
||||||
|
1. Exports package lists (pacman explicit, all packages, AUR packages)
|
||||||
|
2. Exports Docker volumes to compressed tarballs
|
||||||
|
3. Backs up entire `/etc/` directory
|
||||||
|
4. Prunes old backups (keeps: 7 daily, 4 weekly, 6 monthly, 2 yearly)
|
||||||
|
5. Shows backup statistics
|
||||||
|
|
||||||
|
## Restore Procedures
|
||||||
|
|
||||||
|
### List Available Backups
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
|
||||||
|
restic -r /mnt/nas/backups/homelab-restic snapshots
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restore After System Reinstall
|
||||||
|
|
||||||
|
1. **Restore backup to temporary location:**
|
||||||
|
```bash
|
||||||
|
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
|
||||||
|
restic -r /mnt/nas/backups/homelab-restic restore latest --target /tmp/restore
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Reinstall packages:**
|
||||||
|
```bash
|
||||||
|
# Reinstall all explicitly installed packages
|
||||||
|
sudo pacman -S --needed $(cat /tmp/restore/tmp/system-backup-*/pacman-explicit.txt)
|
||||||
|
|
||||||
|
# Reinstall AUR packages
|
||||||
|
yay -S --needed $(cat /tmp/restore/tmp/system-backup-*/aur-packages.txt)
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Restore system configs:**
|
||||||
|
```bash
|
||||||
|
sudo cp -r /tmp/restore/etc/* /etc/
|
||||||
|
sudo systemctl daemon-reload
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Restore Docker volumes:**
|
||||||
|
```bash
|
||||||
|
# Example: Restore Gitea
|
||||||
|
docker-compose -f /opt/docker/gitea/docker-compose.yml down
|
||||||
|
|
||||||
|
sudo tar xzf /tmp/restore/tmp/docker-backup-*/gitea-data.tar.gz -C /tmp/gitea-restore
|
||||||
|
|
||||||
|
docker run --rm -v gitea_gitea:/data -v /tmp/gitea-restore:/backup alpine \
|
||||||
|
sh -c "rm -rf /data/* && cp -a /backup/* /data/"
|
||||||
|
|
||||||
|
docker-compose -f /opt/docker/gitea/docker-compose.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
### Rollback After Broken Update
|
||||||
|
|
||||||
|
If a system update breaks something:
|
||||||
|
|
||||||
|
1. **Restore specific config:**
|
||||||
|
```bash
|
||||||
|
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
|
||||||
|
|
||||||
|
# Restore nginx config
|
||||||
|
restic -r /mnt/nas/backups/homelab-restic restore latest \
|
||||||
|
--target /tmp/restore --include /etc/nginx/
|
||||||
|
|
||||||
|
sudo cp -r /tmp/restore/etc/nginx/* /etc/nginx/
|
||||||
|
sudo systemctl reload nginx
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Downgrade packages:**
|
||||||
|
```bash
|
||||||
|
# Check which packages were updated
|
||||||
|
grep -A5 "upgraded" /var/log/pacman.log | tail -20
|
||||||
|
|
||||||
|
# Downgrade specific package to cached version
|
||||||
|
sudo pacman -U /var/cache/pacman/pkg/package-old-version.pkg.tar.zst
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restore Specific Docker Volume
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
|
||||||
|
|
||||||
|
# List files in backup
|
||||||
|
restic -r /mnt/nas/backups/homelab-restic ls latest | grep docker-backup
|
||||||
|
|
||||||
|
# Restore specific volume tarball
|
||||||
|
restic -r /mnt/nas/backups/homelab-restic restore latest \
|
||||||
|
--target /tmp/restore \
|
||||||
|
--include /tmp/docker-backup-*/nextcloud-data.tar.gz
|
||||||
|
|
||||||
|
# Extract and restore to Docker volume
|
||||||
|
mkdir /tmp/nextcloud-data
|
||||||
|
tar xzf /tmp/restore/tmp/docker-backup-*/nextcloud-data.tar.gz -C /tmp/nextcloud-data
|
||||||
|
|
||||||
|
docker run --rm -v nextcloud_nextcloud_data:/data -v /tmp/nextcloud-data:/backup alpine \
|
||||||
|
sh -c "rm -rf /data/* && cp -a /backup/* /data/"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Backup Retention Policy
|
||||||
|
|
||||||
|
The backup script automatically prunes old backups:
|
||||||
|
- **Daily**: Keep last 7 days
|
||||||
|
- **Weekly**: Keep last 4 weeks
|
||||||
|
- **Monthly**: Keep last 6 months
|
||||||
|
- **Yearly**: Keep last 2 years
|
||||||
|
|
||||||
|
This balances storage space with recovery options.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### "Repository not initialized" Error
|
||||||
|
|
||||||
|
Run the initialization script:
|
||||||
|
```bash
|
||||||
|
sudo /home/hoborg/homelab/scripts/backup-init.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### "Password incorrect" Error
|
||||||
|
|
||||||
|
Check that password file exists and is readable:
|
||||||
|
```bash
|
||||||
|
ls -l /home/hoborg/creds/restic-password.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### Check Backup Size
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
|
||||||
|
restic -r /mnt/nas/backups/homelab-restic stats --mode restore-size
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify Backup Integrity
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
|
||||||
|
restic -r /mnt/nas/backups/homelab-restic check
|
||||||
|
```
|
||||||
|
|
||||||
|
## Security Notes
|
||||||
|
|
||||||
|
- Backups are encrypted with AES-256
|
||||||
|
- Password file has `600` permissions (owner read/write only)
|
||||||
|
- Shadow/gshadow files are excluded from backups
|
||||||
|
- NAS should have access restrictions
|
||||||
|
- Consider offsite backup copy of password file
|
||||||
|
|
||||||
|
## Future Enhancements
|
||||||
|
|
||||||
|
- [ ] Automated systemd timer setup
|
||||||
|
- [ ] Pre-backup notifications
|
||||||
|
- [ ] Post-backup success/failure notifications
|
||||||
|
- [ ] Backup verification script
|
||||||
|
- [ ] Web dashboard for backup status
|
||||||
|
- [ ] Offsite backup replication (optional)
|
||||||
Reference in New Issue
Block a user