homelab/docs/network-security.md

# Network & Security Configuration

Complete guide for securing and networking your homelab.

## SSH Security Setup

### Initial Configuration
```bash
# Generate SSH key pair
ssh-keygen -t ed25519 -C "homelab-key" -f ~/.ssh/homelab_ed25519

# Copy public key to target
ssh-copy-id -i ~/.ssh/homelab_ed25519.pub user@target
```

### SSH Hardening

**Status:** ✅ **Complete** - Port changed to 2222

Edit `/etc/ssh/sshd_config`:
```
# Disable root login
PermitRootLogin no

# Use key-based authentication only
PasswordAuthentication no
PubkeyAuthentication yes

# Change default port (CRITICAL - currently still on 22)
Port 2222

# Restrict users
AllowUsers hoborg

# Security settings
Protocol 2
X11Forwarding no
AllowTcpForwarding no
ClientAliveInterval 300
ClientAliveCountMax 2
MaxAuthTries 3
MaxStartups 2
```

**Completed:**
1. ✅ Changed SSH port from 22 to 2222
2. ✅ Updated router port forwarding rules
3. ✅ External access via ak-homelab.duckdns.org:2222 working

Restart SSH: `sudo systemctl restart sshd`

### Mosh Alternative (Investigation Needed)

**Issue:** SSH can be unreliable on WiFi connections with packet loss.

**Mosh Benefits:**
- Maintains connection during network switches (ethernet ↔ WiFi)
- Handles poor WiFi connections better
- Local echo for responsive typing
- Roaming support (IP changes don't break connection)

**Installation:**
```bash
# Server side
sudo pacman -S mosh

# Client side
mosh user@server
```

**Requirements:**
- UDP ports 60000-61000 open on router
- SSH still needed for initial authentication

**Status:** ✅ **Local use working** ❌ **External blocked by ISP**

**Key Findings:**
- **Local mosh**: Works perfectly (`mosh localhost`, `mosh 192.168.0.100`)
- **External mosh**: Blocked by ISP UDP port filtering on ports 60000-61000
- **SSH still needed**: Mosh uses SSH for initial authentication, then switches to UDP

**ISP UDP Blocking Issue:**
- Most ISPs block UDP ports 60000-61000 for "security"
- SSH works fine (TCP port 2222) but mosh fails (UDP 60000-61000)  
- Router port forwarding is correct, but ISP drops UDP packets

**Current Recommendation:**
- Use mosh for local/internal network connections
- Stick with SSH for external connections until VPN is set up
- VPN tunnel can bypass ISP UDP blocking

### SSH Client Configuration
Create `~/.ssh/config`:
```
Host homelab
    HostName ak-homelab.duckdns.org
    User hoborg
    Port 2222
    IdentityFile ~/.ssh/homelab_ed25519
    ServerAliveInterval 60
```

**Usage:**
```bash
# Connect via SSH
ssh homelab

# Connect via Mosh (uses SSH config automatically)
mosh homelab
```

## Dynamic DNS with DuckDNS

### Account Setup
1. Create account at duckdns.org
2. Create subdomain: `ak-homelab.duckdns.org`
3. Get token from dashboard

### Automatic IP Updates
Update script at `~/.local/scripts/duckdns.py` (Python implementation)

Cron job for automatic updates:
```bash
# Update every 5 minutes
*/5 * * * * /home/hoborg/.local/scripts/duckdns.py >/dev/null 2>&1
```

### Current Setup (Router-based)

**Status:** ✅ **Migrated from script to router DynDNS**

**Changes made:**
- ✅ Disabled cron job script (`*/5 * * * *` entry removed)
- ✅ Enabled router Dynamic DNS for ak-homelab.duckdns.org
- ⏳ **Testing pending** - Cannot force public IP change to verify

**Router DynDNS Benefits:**
- Immediate updates on IP change (vs 5-minute delay)
- Works when server is down
- Lower resource usage

**Limitations:**
- Likely IPv4-only (Sagemcom router limitation)
- Less control over update process

### Testing
```bash
# Check current IP
curl -s https://ipinfo.io/ip

# Verify DNS resolution
nslookup ak-homelab.duckdns.org

# Check IPv6 (likely not updated by router)
nslookup -type=AAAA ak-homelab.duckdns.org
```

**Testing will occur naturally when ISP changes public IP address.**

## VPN Setup with WireGuard

### What is WireGuard?
WireGuard is a modern, lightweight VPN protocol that creates secure tunnels between devices. It encrypts all network traffic and routes it through a VPN server, making your internet connection private and secure.

**Key benefits:**
- **Privacy**: Hides your IP address and encrypts traffic
- **Security**: Protects against man-in-the-middle attacks on public WiFi
- **Access**: Bypass geo-restrictions and enables remote homelab access
- **Performance**: Much faster than OpenVPN with lower battery drain
- **Simplicity**: Easy to configure compared to other VPN protocols

**When you need VPN:**
- Accessing homelab remotely over internet
- Working from public WiFi frequently
- Need to bypass ISP restrictions
- Running public-facing services

**Costs:** WireGuard itself is free. Self-hosted VPN costs $5-20/month for VPS hosting.

**Use cases:**
- Access homelab services remotely (SSH, web interfaces, file shares)
- Secure connection on public WiFi
- Bypass ISP restrictions or geo-blocks

**Performance:** Much faster and lighter than OpenVPN, better battery life on mobile devices.

### Server Configuration
Install WireGuard: `pacman -S wireguard-tools`

Generate keys:
```bash
wg genkey | tee server_private.key | wg pubkey > server_public.key
```

Server config `/etc/wireguard/wg0.conf`:
```ini
[Interface]
PrivateKey = <SERVER_PRIVATE_KEY>
Address = 10.0.0.1/24
ListenPort = 51820
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
PublicKey = <CLIENT_PUBLIC_KEY>
AllowedIPs = 10.0.0.2/32
```

### Client Configuration
Generate client keys:
```bash
wg genkey | tee client_private.key | wg pubkey > client_public.key
```

Client config:
```ini
[Interface]
PrivateKey = <CLIENT_PRIVATE_KEY>
Address = 10.0.0.2/24
DNS = 1.1.1.1

[Peer]
PublicKey = <SERVER_PUBLIC_KEY>
Endpoint = ak-homelab.duckdns.org:51820
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 25
```

### Enable VPN
```bash
sudo systemctl enable wg-quick@wg0
sudo systemctl start wg-quick@wg0
```

## Firewall Configuration

### UFW (Uncomplicated Firewall)

**What it does:** Controls what network traffic is allowed in/out of your server.

**Key functions:**
- **Default deny**: Blocks all incoming connections by default
- **Port control**: Open only specific ports you need (SSH, HTTP, etc.)
- **Rate limiting**: Prevent brute force attacks
- **Application profiles**: Pre-configured rules for common services

**Why needed:** Without firewall, all services are exposed to network attacks.

### UFW Setup
```bash
# Install and enable UFW
pacman -S ufw
sudo ufw enable

# Default policies
sudo ufw default deny incoming
sudo ufw default allow outgoing

# SSH access
sudo ufw allow 2222/tcp

# WireGuard
sudo ufw allow 51820/udp

# HTTP/HTTPS for services
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
```

### Advanced Rules
```bash
# Rate limiting for SSH
sudo ufw limit 2222/tcp

# Allow from specific networks
sudo ufw allow from 192.168.1.0/24 to any port 22

# Log denied connections
sudo ufw logging on
```

## Network Security Best Practices

### Port Management
- **Change default ports**: SSH (2222), WireGuard (51820)
- **Close unused ports**: Regular port scans with nmap
- **Port forwarding**: Only forward necessary ports

### Access Control
- **VPN-first approach**: Access services through VPN tunnel
- **IP whitelisting**: Restrict access to known IPs when possible
- **Rate limiting**: Prevent brute force attacks

### Monitoring
- **Log analysis**: Monitor `/var/log/auth.log` for SSH attempts
- **Network monitoring**: Use netstat/ss to check listening ports
- **Intrusion detection**: Use fail2ban for automated blocking

## fail2ban - Intrusion Prevention

### What is fail2ban?

**What it does:** Automatically blocks IP addresses that show malicious behavior.

**Key functions:**
- **Log monitoring**: Watches system logs for suspicious activity
- **Pattern detection**: Identifies failed login attempts, scanning, etc.
- **Automatic blocking**: Temporarily bans offending IP addresses
- **Customizable rules**: Configure what triggers a ban and for how long

**Common protections:**
- SSH brute force attempts
- Web server attacks (404 scanning, etc.)
- Email server abuse
- Custom application attacks

**Example:** After 5 failed SSH login attempts in 10 minutes, ban IP for 1 hour.

**Why important:** Reduces server load and prevents automated attacks from succeeding through persistence.

### fail2ban Installation & Setup

```bash
# Install fail2ban
sudo pacman -S fail2ban

# Enable and start service
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

# Create local configuration
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
```

### Basic SSH Protection Configuration

Edit `/etc/fail2ban/jail.local`:
```ini
[sshd]
enabled = true
port = 2222
filter = sshd
logpath = /var/log/auth.log
maxretry = 5
bantime = 3600
findtime = 600
```

**Configuration explained:**
- `maxretry = 5`: Ban after 5 failed attempts
- `bantime = 3600`: Ban for 1 hour (3600 seconds)
- `findtime = 600`: 5 attempts within 10 minutes triggers ban
- `port = 2222`: Monitor custom SSH port

### Restart and Monitor

```bash
# Restart fail2ban to apply changes
sudo systemctl restart fail2ban

# Check status
sudo fail2ban-client status
sudo fail2ban-client status sshd

# View banned IPs
sudo fail2ban-client get sshd banned
```

## Geographic IP Blocking with geoip-shell

### Overview
**Status:** ✅ **Implemented** - Whitelist mode protecting all interfaces

**Tool:** geoip-shell v0.7.5
**Repository:** https://github.com/friendly-bits/geoip-shell
**Purpose:** Geographic IP blocking for enhanced security

### Current Configuration
- **Firewall Backend:** iptables
- **IP Lists Source:** RIPE (Réseaux IP Européens)
- **Network Interfaces:** All interfaces protected
- **LAN Detection:** Automatic subnet detection enabled
- **Mode:** Whitelist (only specified countries allowed)
- **IP Families:** IPv4 and IPv6 supported
- **Update Schedule:** Daily at 4:18 AM
- **Last Update:** September 17, 2025 at 00:57:41

### Whitelisted Countries
```
AL (Albania), AD (Andorra), AM (Armenia), AT (Austria), AZ (Azerbaijan)
BY (Belarus), BE (Belgium), BA (Bosnia and Herzegovina), BG (Bulgaria)
HR (Croatia), CY (Cyprus), CZ (Czech Republic), DK (Denmark)
EE (Estonia), FO (Faroe Islands), FI (Finland), FR (France)
GE (Georgia), DE (Germany), GI (Gibraltar), GR (Greece)
GG (Guernsey), HU (Hungary), IS (Iceland), IE (Ireland)
IM (Isle of Man), IT (Italy), JE (Jersey), KZ (Kazakhstan)
LV (Latvia), LI (Liechtenstein), LT (Lithuania), LU (Luxembourg)
MT (Malta), MD (Moldova), MC (Monaco), ME (Montenegro)
NL (Netherlands), MK (North Macedonia), NO (Norway), PL (Poland)
PT (Portugal), RO (Romania), RU (Russia), SM (San Marino)
RS (Serbia), SK (Slovakia), SI (Slovenia), ES (Spain)
SE (Sweden), CH (Switzerland), TR (Turkey), UA (Ukraine)
GB (United Kingdom), VA (Vatican City)
```

### Network Exceptions (Always Allowed)
**IPv4 Networks:**
- `172.18.0.0/16` - Docker network
- `172.17.0.0/16` - Docker network
- `169.254.0.0/16` - Link-local addresses
- `192.168.0.0/24` - Local LAN
- `172.20.0.0/16` - Docker network
- `172.19.0.0/16` - Docker network

**IPv6 Networks:**
- `fdaa:bbcc:ddee::/64` - Custom network
- `fe80::/10` - Link-local addresses

### Security Impact
1. **Geographic Blocking:** Blocks all traffic from countries not in whitelist
2. **Comprehensive Coverage:** Both IPv4 and IPv6 protection
3. **Full Protocol Coverage:** TCP and UDP traffic controlled
4. **Network Awareness:** Automatically detects and allows local networks
5. **Persistence:** Rules survive system reboots
6. **Automatic Updates:** IP lists updated daily

### Integration with Existing Security
- **Complements fail2ban:** Provides geographic layer above intrusion detection
- **Works with UFW:** Uses iptables backend compatible with UFW
- **Docker Compatible:** Automatically allows Docker networks
- **LAN Friendly:** Preserves local network access

### Monitoring
```bash
# Check geoip-shell status
geoip-shell status

# View logs
journalctl -u geoip-shell
tail -f /var/log/geoip-shell.log
```

### Manual Setup Process
The tool was installed manually with interactive prompts rather than scripted installation due to security considerations. Key decisions made during setup:

1. **Whitelist Mode:** Chosen over blacklist for better control
2. **European Focus:** Primary whitelist consists of European countries
3. **Network Exceptions:** Docker and LAN networks automatically detected
4. **Dual Stack:** Both IPv4 and IPv6 protection enabled
5. **Full Protocol Coverage:** TCP and UDP both protected

### Detailed Documentation
For complete setup details, see **[docs/geoip-blocking.md](docs/geoip-blocking.md)**

## Permanent Ban System for Repeat Offenders

### Overview
**Status:** ✅ **Implemented** - Automated permanent banning of persistent attackers

**Script:** `scripts/permanent-ban-repeat-offenders.sh`
**Purpose:** Automatically identify and permanently ban IPs that have been banned by fail2ban more than a threshold number of times

### How It Works

#### Detection Logic
1. **Log Analysis:** Scans `/var/log/fail2ban.log*` for ban entries
2. **IP Extraction:** Extracts IP addresses from ban log entries
3. **Frequency Counting:** Counts how many times each IP has been banned
4. **Threshold Check:** Identifies IPs banned more than the threshold (4 times)

#### Permanent Banning Process
For each repeat offender:
1. **Country Lookup:** Uses `whois` to determine the country of origin
2. **Banlist Update:** Adds IP to `/etc/fail2ban/permanent-banlist.conf`
3. **Firewall Rule:** Creates permanent iptables DROP rule
4. **Persistence:** Saves iptables rules to `/etc/iptables/iptables.rules`
5. **Service Reload:** Reloads fail2ban to recognize the updated banlist

### Configuration

#### Threshold Settings
```bash
THRESHOLD=4  # Ban after 4 fail2ban bans
```

#### File Locations
- **Log File:** `/var/log/permanent-ban.log`
- **Banlist:** `/etc/fail2ban/permanent-banlist.conf`
- **Iptables Rules:** `/etc/iptables/iptables.rules`

#### Cron Schedule
- **Frequency:** Every 6 hours (`0 */6 * * *`)
- **User:** root
- **Command:** `/home/hoborg/homelab/scripts/permanent-ban-repeat-offenders.sh`

### Security Benefits

#### Multi-Layer Protection
1. **fail2ban:** Temporary bans for suspicious activity
2. **Permanent Bans:** Long-term blocking of persistent attackers
3. **Geographic Blocking:** Country-level filtering via geoip-shell
4. **Network-Level:** iptables rules at the firewall level

#### Attack Prevention
- **Brute Force:** Blocks IPs that repeatedly attempt attacks
- **Botnets:** Prevents automated attacks from compromised hosts
- **Persistence:** Maintains bans across system reboots
- **Resource Protection:** Reduces server load from repeat offenders

### Monitoring and Maintenance

#### Log Analysis
```bash
# View permanent ban activity
tail -f /var/log/permanent-ban.log

# Check current permanent bans
cat /etc/fail2ban/permanent-banlist.conf

# View iptables permanent rules
iptables -L | grep DROP
```

#### Manual Execution
```bash
# Run the script manually (requires root)
sudo /home/hoborg/homelab/scripts/permanent-ban-repeat-offenders.sh
```

#### Unban Procedure
To manually unban a permanently banned IP:
```bash
# Remove from banlist
sudo sed -i "/^192\.168\.1\.100/d" /etc/fail2ban/permanent-banlist.conf

# Remove iptables rule (find the rule number first)
sudo iptables -L --line-numbers | grep "192.168.1.100"
sudo iptables -D INPUT <rule_number>

# Save iptables rules
sudo iptables-save > /etc/iptables/iptables.rules

# Reload fail2ban
sudo systemctl reload fail2ban
```

### Integration with Security Stack

#### Complementary Tools
- **fail2ban:** Provides temporary bans that feed into permanent ban detection
- **geoip-shell:** Geographic blocking at the network level
- **UFW:** Additional firewall layer
- **SSH Hardening:** Reduces initial attack surface

#### Workflow
```
Attack Attempt → fail2ban Detection → Temporary Ban → Repeat Offense → Permanent Ban → Geographic Block
```

### Troubleshooting

#### Common Issues
- **Script Not Running:** Check cron job configuration
- **Permission Errors:** Ensure script is executable and paths are correct
- **whois Failures:** Some IPs may not return country information
- **iptables-save Issues:** Check if iptables-persistent is installed

#### Diagnostic Commands
```bash
# Check cron service
sudo systemctl status cron

# Test script manually
sudo bash -x /home/hoborg/homelab/scripts/permanent-ban-repeat-offenders.sh

# Verify iptables rules
sudo iptables -L -n | grep DROP

# Check fail2ban integration
sudo fail2ban-client status
```

## Router Configuration

### Port Forwarding
Forward these ports to your homelab server:
- SSH: External port → Internal 2222
- WireGuard: 51820 → 51820
- Web services: 80/443 → 80/443 (if needed)

### Security Settings
- **Disable WPS**: Turn off WiFi Protected Setup
- **Strong WiFi password**: WPA3 with complex passphrase
- **Guest network**: Separate network for guests
- **Firmware updates**: Keep router firmware current

## Network Planning

### Dual Network Interface Issue (Critical)

**Problem:** Server has both ethernet and WiFi interfaces. When switching between connections, IP address changes from ethernet (192.168.0.22) to different WiFi IP, breaking SSH connections and port forwards.

**Limitation:** Most routers don't allow DHCP reservation of same IP for multiple MAC addresses.

**Solutions:**

**Option 1: Static IP Configuration (Recommended)**
Configure both interfaces with same static IP:
```bash
# Check interface names
ip link show

# Configure ethernet interface
sudo systemctl edit --full systemd-networkd
# Create /etc/systemd/network/20-ethernet.network
[Match]
Name=enp*

[Network]
DHCP=no
Address=192.168.0.100/24
Gateway=192.168.0.1
DNS=192.168.0.1

# Create /etc/systemd/network/25-wifi.network
[Match]
Name=wlp*

[Network]
DHCP=no
Address=192.168.0.100/24
Gateway=192.168.0.1
DNS=192.168.0.1
```

**Option 2: Hostname-based Access**
Use local hostname resolution instead of IP:
```bash
# Access via hostname (works for both interfaces)
ssh hoborg@ak-homelab.local
# or configure local DNS/mDNS
```

**Option 3: Bridge Networking** 
Create bridge combining both interfaces for automatic failover:
```bash
# Advanced: Bridge both interfaces
ip link add name br0 type bridge
ip link set enp3s0 master br0
ip link set wlp2s0 master br0
```

**Current Setup:**
- Router: 192.168.0.1
- Ethernet: 192.168.0.100 (static IP achieved)
- WiFi: Static IP needed (same .100)
- External: ak-homelab.duckdns.org ✅
- SSH: Port 2222 ✅

## Network Boot Issues Troubleshooting

### Problem 1: Network not available after reboot until GUI login

**Initial diagnosis**: Thought to be NetworkManager sleep behavior.

**Actual root cause**: System auto-suspend due to closed laptop lid (`HandleLidSwitch=suspend`).

**Symptoms**:
- SSH inaccessible after reboot 
- Network comes up only when laptop lid is opened
- System logs show "Suspending..." followed by "Lid opened"

**Solution Applied**:
1. **systemd-logind configuration**: Disable lid switch handling for headless server operation
2. **Static IP configuration**: Eliminate DHCP negotiation delays  
3. **High connection priority**: Ensure ethernet connects first

**Configuration files created**:

`/etc/systemd/logind.conf.d/01-server-logind.conf`:
```ini
[Login]
# Disable all power management triggers for headless server
HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore  
HandleLidSwitchDocked=ignore

# Prevent automatic suspend/hibernate
IdleAction=ignore

# Keep system running even when no users logged in
KillUserProcesses=no
```

**NetworkManager commands**:
```bash
nmcli connection modify "Wired connection 2" connection.autoconnect-priority 10
nmcli connection modify "Wired connection 2" ipv4.method manual ipv4.addresses 192.168.0.100/24 ipv4.gateway 192.168.0.1 ipv4.dns "84.2.44.8 84.2.46.8"
```

**Deployment**:
```bash
sudo cp config/systemd/01-server-logind.conf /etc/systemd/logind.conf.d/
sudo systemctl restart systemd-logind
sudo cp config/networkmanager/01-homelab.conf /etc/NetworkManager/conf.d/
sudo systemctl reload NetworkManager
```

### Diagnostic Commands

**Check system power state**:
```bash
systemctl status systemd-logind  # Check for suspend/lid events
loginctl show-session           # Current power management settings
```

**Check network connectivity**:
```bash
systemctl status NetworkManager
nmcli device status
nmcli connection show "Wired connection 2"
ip addr show enp4s0
```

**Monitor boot process**:
```bash
journalctl -b -u NetworkManager    # Network startup logs
journalctl -b -u sshd             # SSH service logs  
journalctl -b -u systemd-logind   # Power management events
```

**Result**: ✅ **RESOLVED** - SSH accessible immediately on boot, lid closure no longer suspends system.

**Network Interface Identification:**
- **enp3s0f0**: First ethernet port (98:fa:9b:f1:06:d5)
- **enp4s0**: Second ethernet port (98:fa:9b:f1:06:d4) ← **Use this one**
- **wlp1s0**: WiFi interface (0c:dd:24:e6:0f:87)

**Issue Solved:** Dual ethernet ports caused MAC address confusion when cable was moved between ports. Stick to enp4s0 consistently.

### IP Address Scheme
- **Router**: 192.168.0.1
- **Homelab server**: 192.168.0.100 (target static IP)
- **Current ethernet**: 192.168.0.22 (can migrate to .100)
- **DHCP range**: 192.168.0.10-99 (excluding static IPs)
- **VPN subnet**: 10.0.0.0/24

### DNS Configuration
- **Primary DNS**: Router (192.168.1.1)
- **Secondary DNS**: 1.1.1.1, 8.8.8.8
- **Local domain**: homelab.local
- **Dynamic DNS**: ak-homelab.duckdns.org

### Service Architecture
```
Internet → Router → Homelab Server
                 ↓
    ┌─────────────────────────────┐
    │  SSH (2222)                 │
    │  WireGuard VPN (51820)      │
    │  Web Services (80/443)      │
    │  Monitoring & Logging       │
    └─────────────────────────────┘
```

## Backup and Recovery

### Configuration Backups
- **SSH keys**: Store securely, separate from server
- **WireGuard configs**: Document peer configurations
- **Firewall rules**: Export UFW rules with `ufw status numbered`

### Network Documentation
- **IP mappings**: Document static assignments
- **Port forwards**: List all forwarded ports and purposes
- **Access credentials**: Secure storage of passwords/keys