Fix SSL certificate renewal for Let's Encrypt

Allow ACME challenge requests before HTTPS redirect: - Add location block for /.well-known/acme-challenge/ - Prevent immediate redirect that was blocking Let's Encrypt verification - Fixes 'Timeout during connect' errors during certbot renewal
Document voice assistant TTS service status
2025-11-15 22:30:01 +01:00 · 2025-10-11 18:25:08 +02:00 · 2025-10-11 18:24:58 +02:00 · 2025-10-11 18:24:51 +02:00 · 2025-10-11 18:24:41 +02:00
6 changed files with 543 additions and 3 deletions
--- a/config/nginx/homelab.conf
+++ b/config/nginx/homelab.conf
@@ -4,9 +4,16 @@
 server {
    listen 80 default_server;
    server_name ak-homelab.duckdns.org _;
-    
-    # Redirect HTTP to HTTPS
-    return 301 https://$host$request_uri;
+
+    # Allow Let's Encrypt ACME challenges
+    location /.well-known/acme-challenge/ {
+        root /var/www/certbot;
+    }
+
+    # Redirect all other HTTP to HTTPS
+    location / {
+        return 301 https://$host$request_uri;
+    }
 }

 server {
--- a/docs/system-backups.md
+++ b/docs/system-backups.md
@@ -0,0 +1,233 @@
+# System Backups
+
+Complete backup strategy for the homelab using Restic for encrypted, deduplicated backups to NAS.
+
+## Overview
+
+**Backup Tool**: Restic (modern, encrypted, deduplicated)
+**Backup Destination**: NAS at `/mnt/nas/backups/homelab-restic`
+**Backup Frequency**: Daily (can be automated with systemd timer)
+**Encryption**: AES-256, password stored in `/home/hoborg/creds/restic-password.txt`
+
+## What Gets Backed Up
+
+### Full System Backup Includes:
+1. **Package Lists**
+   - All explicitly installed packages (pacman)
+   - All installed packages (including dependencies)
+   - AUR/foreign packages list
+   - Allows exact system recreation after reinstall
+
+2. **System Configuration** (`/etc/`)
+   - Nginx configuration
+   - Systemd services
+   - Network configuration
+   - All system-wide configs
+   - Excludes: shadow/gshadow password files
+
+3. **Docker Volumes**
+   - Gitea (Git repositories and database)
+   - Jellyfin (media library configuration)
+   - Nextcloud (cloud storage data and database)
+   - Portainer (container management data)
+
+### What's NOT Backed Up (Intentionally):
+- **User data in `/home`**: Separate partition, safe from system reinstalls
+- **Media files**: Already stored on NAS (Music, Videos, Pictures)
+- **User documents**: Synced via Nextcloud/Copyparty or on separate partition
+
+## Setup
+
+### Initial Setup (One-Time)
+
+1. **Initialize the backup repository:**
+   ```bash
+   sudo /home/hoborg/homelab/scripts/backup-init.sh
+   ```
+
+   This will:
+   - Install Restic if needed
+   - Create backup directory on NAS (`/mnt/nas/backups/homelab-restic`)
+   - Generate encryption password (saved to `/home/hoborg/creds/restic-password.txt`)
+   - Initialize the Restic repository
+
+   ⚠️ **IMPORTANT**: Back up the password file! Without it, backups cannot be restored.
+
+### Running Backups
+
+**Manual backup:**
+```bash
+sudo /home/hoborg/homelab/scripts/backup-system-full.sh
+```
+
+**Automated backups** (optional - setup systemd timer later):
+- Create systemd service and timer
+- Schedule daily backups (e.g., 3 AM)
+- Logs written to `/var/log/homelab-backup.log`
+
+## Backup Scripts
+
+### `backup-init.sh`
+Initializes the Restic repository. Run once during setup.
+
+Location: `/home/hoborg/homelab/scripts/backup-init.sh`
+
+### `backup-system-full.sh`
+Performs full system backup including packages, configs, and Docker volumes.
+
+Location: `/home/hoborg/homelab/scripts/backup-system-full.sh`
+
+What it does:
+1. Exports package lists (pacman explicit, all packages, AUR packages)
+2. Exports Docker volumes to compressed tarballs
+3. Backs up entire `/etc/` directory
+4. Prunes old backups (keeps: 7 daily, 4 weekly, 6 monthly, 2 yearly)
+5. Shows backup statistics
+
+## Restore Procedures
+
+### List Available Backups
+
+```bash
+export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
+restic -r /mnt/nas/backups/homelab-restic snapshots
+```
+
+### Restore After System Reinstall
+
+1. **Restore backup to temporary location:**
+   ```bash
+   export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
+   restic -r /mnt/nas/backups/homelab-restic restore latest --target /tmp/restore
+   ```
+
+2. **Reinstall packages:**
+   ```bash
+   # Reinstall all explicitly installed packages
+   sudo pacman -S --needed $(cat /tmp/restore/tmp/system-backup-*/pacman-explicit.txt)
+
+   # Reinstall AUR packages
+   yay -S --needed $(cat /tmp/restore/tmp/system-backup-*/aur-packages.txt)
+   ```
+
+3. **Restore system configs:**
+   ```bash
+   sudo cp -r /tmp/restore/etc/* /etc/
+   sudo systemctl daemon-reload
+   ```
+
+4. **Restore Docker volumes:**
+   ```bash
+   # Example: Restore Gitea
+   docker-compose -f /opt/docker/gitea/docker-compose.yml down
+
+   sudo tar xzf /tmp/restore/tmp/docker-backup-*/gitea-data.tar.gz -C /tmp/gitea-restore
+
+   docker run --rm -v gitea_gitea:/data -v /tmp/gitea-restore:/backup alpine \
+     sh -c "rm -rf /data/* && cp -a /backup/* /data/"
+
+   docker-compose -f /opt/docker/gitea/docker-compose.yml up -d
+   ```
+
+### Rollback After Broken Update
+
+If a system update breaks something:
+
+1. **Restore specific config:**
+   ```bash
+   export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
+
+   # Restore nginx config
+   restic -r /mnt/nas/backups/homelab-restic restore latest \
+     --target /tmp/restore --include /etc/nginx/
+
+   sudo cp -r /tmp/restore/etc/nginx/* /etc/nginx/
+   sudo systemctl reload nginx
+   ```
+
+2. **Downgrade packages:**
+   ```bash
+   # Check which packages were updated
+   grep -A5 "upgraded" /var/log/pacman.log | tail -20
+
+   # Downgrade specific package to cached version
+   sudo pacman -U /var/cache/pacman/pkg/package-old-version.pkg.tar.zst
+   ```
+
+### Restore Specific Docker Volume
+
+```bash
+export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
+
+# List files in backup
+restic -r /mnt/nas/backups/homelab-restic ls latest | grep docker-backup
+
+# Restore specific volume tarball
+restic -r /mnt/nas/backups/homelab-restic restore latest \
+  --target /tmp/restore \
+  --include /tmp/docker-backup-*/nextcloud-data.tar.gz
+
+# Extract and restore to Docker volume
+mkdir /tmp/nextcloud-data
+tar xzf /tmp/restore/tmp/docker-backup-*/nextcloud-data.tar.gz -C /tmp/nextcloud-data
+
+docker run --rm -v nextcloud_nextcloud_data:/data -v /tmp/nextcloud-data:/backup alpine \
+  sh -c "rm -rf /data/* && cp -a /backup/* /data/"
+```
+
+## Backup Retention Policy
+
+The backup script automatically prunes old backups:
+- **Daily**: Keep last 7 days
+- **Weekly**: Keep last 4 weeks
+- **Monthly**: Keep last 6 months
+- **Yearly**: Keep last 2 years
+
+This balances storage space with recovery options.
+
+## Troubleshooting
+
+### "Repository not initialized" Error
+
+Run the initialization script:
+```bash
+sudo /home/hoborg/homelab/scripts/backup-init.sh
+```
+
+### "Password incorrect" Error
+
+Check that password file exists and is readable:
+```bash
+ls -l /home/hoborg/creds/restic-password.txt
+```
+
+### Check Backup Size
+
+```bash
+export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
+restic -r /mnt/nas/backups/homelab-restic stats --mode restore-size
+```
+
+### Verify Backup Integrity
+
+```bash
+export RESTIC_PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
+restic -r /mnt/nas/backups/homelab-restic check
+```
+
+## Security Notes
+
+- Backups are encrypted with AES-256
+- Password file has `600` permissions (owner read/write only)
+- Shadow/gshadow files are excluded from backups
+- NAS should have access restrictions
+- Consider offsite backup copy of password file
+
+## Future Enhancements
+
+- [ ] Automated systemd timer setup
+- [ ] Pre-backup notifications
+- [ ] Post-backup success/failure notifications
+- [ ] Backup verification script
+- [ ] Web dashboard for backup status
+- [ ] Offsite backup replication (optional)
--- a/docs/voice-assistant.md
+++ b/docs/voice-assistant.md
@@ -1,5 +1,7 @@
 # Voice Assistant Setup

+⚠️ **STATUS: DISABLED** - onnxruntime package was removed to free disk space (1.2GB). Voice functionality is currently unavailable.
+
 This document describes how to set up AI voice capabilities for Claude Code using local TTS (Text-to-Speech) services.

 ## Overview
@@ -8,6 +10,7 @@ The voice assistant setup uses:
 - **Piper TTS**: Local neural text-to-speech engine for generating natural-sounding speech
 - **FastAPI**: HTTP server wrapper to make Piper compatible with voice-mode
 - **Ryan voice model**: Professional male US English voice for AI assistant personality
+- **onnxruntime**: ML inference library (removed - required for TTS)

 ## Prerequisites

--- a/scripts/backup-homelab.sh
+++ b/scripts/backup-homelab.sh
@@ -0,0 +1,95 @@
+#!/bin/bash
+# Backup Docker volumes and system configs to NAS using Restic
+# Can be run manually or via systemd timer
+
+set -e
+
+# Configuration
+BACKUP_REPO="/mnt/nas/backups/homelab-restic"
+PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
+LOG_FILE="/var/log/homelab-backup.log"
+DOCKER_BACKUP_DIR="/tmp/docker-backup-$$"
+
+# Export password
+export RESTIC_PASSWORD_FILE="$PASSWORD_FILE"
+
+# Logging function
+log() {
+    echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
+}
+
+log "========================================"
+log "Starting Homelab Backup"
+log "========================================"
+
+# Check if repository exists
+if ! restic -r "$BACKUP_REPO" snapshots &>/dev/null; then
+    log "ERROR: Backup repository not initialized. Run backup-init.sh first."
+    exit 1
+fi
+
+# Create temporary directory for Docker volume exports
+mkdir -p "$DOCKER_BACKUP_DIR"
+log "Created temporary backup directory: $DOCKER_BACKUP_DIR"
+
+# Export Docker volumes
+log "Exporting Docker volumes..."
+
+# Gitea
+log "  - Exporting Gitea data..."
+docker run --rm -v gitea_gitea:/data -v "$DOCKER_BACKUP_DIR":/backup alpine \
+    tar czf /backup/gitea-data.tar.gz -C /data .
+
+# Jellyfin
+log "  - Exporting Jellyfin config..."
+docker run --rm -v jellyfin_jellyfin_config:/data -v "$DOCKER_BACKUP_DIR":/backup alpine \
+    tar czf /backup/jellyfin-config.tar.gz -C /data .
+
+# Nextcloud data
+log "  - Exporting Nextcloud data..."
+docker run --rm -v nextcloud_nextcloud_data:/data -v "$DOCKER_BACKUP_DIR":/backup alpine \
+    tar czf /backup/nextcloud-data.tar.gz -C /data .
+
+# Nextcloud database
+log "  - Exporting Nextcloud database..."
+docker run --rm -v nextcloud_nextcloud_db:/data -v "$DOCKER_BACKUP_DIR":/backup alpine \
+    tar czf /backup/nextcloud-db.tar.gz -C /data .
+
+# Portainer
+log "  - Exporting Portainer data..."
+docker run --rm -v portainer_data:/data -v "$DOCKER_BACKUP_DIR":/backup alpine \
+    tar czf /backup/portainer-data.tar.gz -C /data .
+
+log "✓ Docker volumes exported"
+
+# Run backup
+log "Running Restic backup..."
+restic -r "$BACKUP_REPO" backup \
+    --verbose \
+    --tag homelab-docker \
+    --tag homelab-configs \
+    /etc/nginx/sites-available/homelab \
+    /etc/systemd/system/copyparty.service \
+    /home/hoborg/.config/copyparty/ \
+    "$DOCKER_BACKUP_DIR"
+
+# Cleanup temporary files
+log "Cleaning up temporary files..."
+rm -rf "$DOCKER_BACKUP_DIR"
+
+# Prune old backups (keep: 7 daily, 4 weekly, 6 monthly, 2 yearly)
+log "Pruning old backups..."
+restic -r "$BACKUP_REPO" forget \
+    --keep-daily 7 \
+    --keep-weekly 4 \
+    --keep-monthly 6 \
+    --keep-yearly 2 \
+    --prune
+
+# Show backup stats
+log "Backup statistics:"
+restic -r "$BACKUP_REPO" stats --mode restore-size
+
+log "========================================"
+log "Backup completed successfully!"
+log "========================================"
--- a/scripts/backup-init.sh
+++ b/scripts/backup-init.sh
@@ -0,0 +1,74 @@
+#!/bin/bash
+# Initialize Restic backup repository on NAS
+# Run once to set up the backup system
+
+set -e
+
+# Configuration
+BACKUP_REPO="/mnt/nas/backups/homelab-restic"
+PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
+
+echo "========================================"
+echo "Restic Backup Repository Initialization"
+echo "========================================"
+echo ""
+echo "This will:"
+echo "  1. Install restic if needed"
+echo "  2. Create backup repository at: $BACKUP_REPO"
+echo "  3. Generate and save encryption password"
+echo ""
+
+# Check if restic is installed
+if ! command -v restic &> /dev/null; then
+    echo "Installing restic..."
+    pacman -S --needed --noconfirm restic
+    echo "✓ Restic installed"
+else
+    echo "✓ Restic already installed ($(restic version))"
+fi
+
+# Create backup directory on NAS
+echo ""
+echo "Creating backup directory on NAS..."
+mkdir -p "$BACKUP_REPO"
+echo "✓ Directory created: $BACKUP_REPO"
+
+# Generate random password if doesn't exist
+echo ""
+if [ -f "$PASSWORD_FILE" ]; then
+    echo "✓ Password file already exists: $PASSWORD_FILE"
+else
+    echo "Generating encryption password..."
+    mkdir -p /home/hoborg/creds
+    openssl rand -base64 32 > "$PASSWORD_FILE"
+    chmod 600 "$PASSWORD_FILE"
+    echo "✓ Password saved to: $PASSWORD_FILE"
+    echo "⚠️  IMPORTANT: Back up this password file! Without it, backups cannot be restored."
+fi
+
+# Export password for restic
+export RESTIC_PASSWORD_FILE="$PASSWORD_FILE"
+
+# Initialize repository
+echo ""
+echo "Initializing Restic repository..."
+if restic -r "$BACKUP_REPO" snapshots &>/dev/null; then
+    echo "✓ Repository already initialized"
+else
+    restic -r "$BACKUP_REPO" init
+    echo "✓ Repository initialized"
+fi
+
+echo ""
+echo "========================================"
+echo "Backup Repository Ready!"
+echo "========================================"
+echo ""
+echo "Repository location: $BACKUP_REPO"
+echo "Password file: $PASSWORD_FILE"
+echo ""
+echo "Next steps:"
+echo "  1. Run backup-homelab.sh to create first backup"
+echo "  2. Test restore with backup-restore.sh"
+echo "  3. Enable automated backups with systemd timer"
+echo ""
--- a/scripts/backup-system-full.sh
+++ b/scripts/backup-system-full.sh
@@ -0,0 +1,128 @@
+#!/bin/bash
+# Full system backup - packages, configs, and Docker volumes
+# Can be run manually or via systemd timer
+
+set -e
+
+# Configuration
+BACKUP_REPO="/mnt/nas/backups/homelab-restic"
+PASSWORD_FILE="/home/hoborg/creds/restic-password.txt"
+LOG_FILE="/var/log/homelab-backup.log"
+DOCKER_BACKUP_DIR="/tmp/docker-backup-$$"
+SYSTEM_BACKUP_DIR="/tmp/system-backup-$$"
+
+# Export password
+export RESTIC_PASSWORD_FILE="$PASSWORD_FILE"
+
+# Logging function
+log() {
+    echo "[$(date +'%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
+}
+
+log "========================================"
+log "Starting Full System Backup"
+log "========================================"
+
+# Check if repository exists
+if ! restic -r "$BACKUP_REPO" snapshots &>/dev/null; then
+    log "ERROR: Backup repository not initialized. Run backup-init.sh first."
+    exit 1
+fi
+
+# Create temporary directories
+mkdir -p "$DOCKER_BACKUP_DIR"
+mkdir -p "$SYSTEM_BACKUP_DIR"
+log "Created temporary directories"
+
+# ============================================
+# 1. Export Package Lists
+# ============================================
+log "Exporting package lists..."
+pacman -Qqe > "$SYSTEM_BACKUP_DIR/pacman-explicit.txt"  # Explicitly installed packages
+pacman -Qq > "$SYSTEM_BACKUP_DIR/pacman-all.txt"        # All installed packages
+pacman -Qqm > "$SYSTEM_BACKUP_DIR/aur-packages.txt"     # AUR/foreign packages only
+log "✓ Package lists exported ($(wc -l < $SYSTEM_BACKUP_DIR/pacman-explicit.txt) explicit, $(wc -l < $SYSTEM_BACKUP_DIR/aur-packages.txt) AUR)"
+
+# ============================================
+# 2. Export Docker Volumes
+# ============================================
+log "Exporting Docker volumes..."
+
+# Gitea
+log "  - Exporting Gitea data..."
+docker run --rm -v gitea_gitea:/data -v "$DOCKER_BACKUP_DIR":/backup alpine \
+    tar czf /backup/gitea-data.tar.gz -C /data .
+
+# Jellyfin
+log "  - Exporting Jellyfin config..."
+docker run --rm -v jellyfin_jellyfin_config:/data -v "$DOCKER_BACKUP_DIR":/backup alpine \
+    tar czf /backup/jellyfin-config.tar.gz -C /data .
+
+# Nextcloud data
+log "  - Exporting Nextcloud data..."
+docker run --rm -v nextcloud_nextcloud_data:/data -v "$DOCKER_BACKUP_DIR":/backup alpine \
+    tar czf /backup/nextcloud-data.tar.gz -C /data .
+
+# Nextcloud database
+log "  - Exporting Nextcloud database..."
+docker run --rm -v nextcloud_nextcloud_db:/data -v "$DOCKER_BACKUP_DIR":/backup alpine \
+    tar czf /backup/nextcloud-db.tar.gz -C /data .
+
+# Portainer
+log "  - Exporting Portainer data..."
+docker run --rm -v portainer_data:/data -v "$DOCKER_BACKUP_DIR":/backup alpine \
+    tar czf /backup/portainer-data.tar.gz -C /data .
+
+log "✓ Docker volumes exported"
+
+# ============================================
+# 3. Run Backup
+# ============================================
+log "Running Restic backup..."
+restic -r "$BACKUP_REPO" backup \
+    --verbose \
+    --tag full-system \
+    --exclude '/etc/shadow*' \
+    --exclude '/etc/gshadow*' \
+    --exclude '/etc/passwd-' \
+    --exclude '/etc/group-' \
+    /etc/ \
+    "$DOCKER_BACKUP_DIR" \
+    "$SYSTEM_BACKUP_DIR"
+
+# ============================================
+# 4. Cleanup
+# ============================================
+log "Cleaning up temporary files..."
+rm -rf "$DOCKER_BACKUP_DIR"
+rm -rf "$SYSTEM_BACKUP_DIR"
+
+# ============================================
+# 5. Prune Old Backups
+# ============================================
+log "Pruning old backups..."
+restic -r "$BACKUP_REPO" forget \
+    --keep-daily 7 \
+    --keep-weekly 4 \
+    --keep-monthly 3 \
+    --keep-yearly 1 \
+    --prune
+
+# ============================================
+# 6. Show Stats
+# ============================================
+log "Backup statistics:"
+restic -r "$BACKUP_REPO" stats --mode restore-size
+
+log "========================================"
+log "Full system backup completed!"
+log "========================================"
+log ""
+log "Backed up:"
+log "  - /etc/ (all system configs)"
+log "  - Docker volumes (Gitea, Jellyfin, Nextcloud, Portainer)"
+log "  - Package lists (pacman + AUR)"
+log ""
+log "To restore packages after reinstall:"
+log "  pacman -S --needed \$(cat pacman-explicit.txt)"
+log "  yay -S --needed \$(cat aur-packages.txt)"
Author	SHA1	Message	Date
Arpad Krejczinger	059daa77aa	Fix SSL certificate renewal for Let's Encrypt Allow ACME challenge requests before HTTPS redirect: - Add location block for /.well-known/acme-challenge/ - Prevent immediate redirect that was blocking Let's Encrypt verification - Fixes 'Timeout during connect' errors during certbot renewal	2025-11-15 22:30:01 +01:00
Arpad Krejczinger	9aa881d895	Document voice assistant TTS service status Mark TTS functionality as disabled due to onnxruntime removal (freed 1.2GB disk space during cleanup)	2025-10-11 18:25:08 +02:00
Arpad Krejczinger	ab4a4cfc9c	Add comprehensive backup system documentation Document complete backup strategy covering: - Setup and initialization procedures - What gets backed up (and what doesn't) - Multiple restore scenarios (full reinstall, rollback, specific volumes) - Retention policy explanation - Troubleshooting and security notes	2025-10-11 18:24:58 +02:00
Arpad Krejczinger	646d3e59bc	Add comprehensive system backup scripts Implement full system backup with Restic including: - Docker volume exports (Gitea, Jellyfin, Nextcloud, Portainer) - System configuration backup (/etc/) - Package list exports (pacman explicit, all, AUR) - Automated retention policy (7 daily, 4 weekly, 3 monthly, 1 yearly) - Separate temporary directories for Docker and system data	2025-10-11 18:24:51 +02:00
Arpad Krejczinger	ad7270aa74	Add Restic backup repository initialization script Initialize encrypted backup repository on NAS with: - Auto-install of Restic if needed - Secure password generation and storage - Repository initialization with AES-256 encryption	2025-10-11 18:24:41 +02:00