homelab/docs/network-security.md

Network & Security Configuration

Complete guide for securing and networking your homelab.

SSH Security Setup

Initial Configuration

# Generate SSH key pair
ssh-keygen -t ed25519 -C "homelab-key" -f ~/.ssh/homelab_ed25519

# Copy public key to target
ssh-copy-id -i ~/.ssh/homelab_ed25519.pub user@target

SSH Hardening

Status: ✅ Complete - Port changed to 2222

Edit /etc/ssh/sshd_config:

# Disable root login
PermitRootLogin no

# Use key-based authentication only
PasswordAuthentication no
PubkeyAuthentication yes

# Change default port (CRITICAL - currently still on 22)
Port 2222

# Restrict users
AllowUsers hoborg

# Security settings
Protocol 2
X11Forwarding no
AllowTcpForwarding no
ClientAliveInterval 300
ClientAliveCountMax 2
MaxAuthTries 3
MaxStartups 2

Completed:

1. ✅ Changed SSH port from 22 to 2222
2. ✅ Updated router port forwarding rules
3. ✅ External access via ak-homelab.duckdns.org:2222 working

Restart SSH: sudo systemctl restart sshd

Mosh Alternative (Investigation Needed)

Issue: SSH can be unreliable on WiFi connections with packet loss.

Mosh Benefits:

• Maintains connection during network switches (ethernet ↔ WiFi)
• Handles poor WiFi connections better
• Local echo for responsive typing
• Roaming support (IP changes don't break connection)

Installation:

# Server side
sudo pacman -S mosh

# Client side
mosh user@server

Requirements:

• UDP ports 60000-61000 open on router
• SSH still needed for initial authentication

Status: ✅ Local use working ❌ External blocked by ISP

Key Findings:

• Local mosh: Works perfectly (mosh localhost, mosh 192.168.0.100)
• External mosh: Blocked by ISP UDP port filtering on ports 60000-61000
• SSH still needed: Mosh uses SSH for initial authentication, then switches to UDP

ISP UDP Blocking Issue:

• Most ISPs block UDP ports 60000-61000 for "security"
• SSH works fine (TCP port 2222) but mosh fails (UDP 60000-61000)
• Router port forwarding is correct, but ISP drops UDP packets

Current Recommendation:

• Use mosh for local/internal network connections
• Stick with SSH for external connections until VPN is set up
• VPN tunnel can bypass ISP UDP blocking

SSH Client Configuration

Create ~/.ssh/config:

Host homelab
    HostName ak-homelab.duckdns.org
    User hoborg
    Port 2222
    IdentityFile ~/.ssh/homelab_ed25519
    ServerAliveInterval 60

Usage:

# Connect via SSH
ssh homelab

# Connect via Mosh (uses SSH config automatically)
mosh homelab

Dynamic DNS with DuckDNS

Account Setup

1. Create account at duckdns.org
2. Create subdomain: ak-homelab.duckdns.org
3. Get token from dashboard

Automatic IP Updates

Update script at ~/.local/scripts/duckdns.py (Python implementation)

Cron job for automatic updates:

# Update every 5 minutes
*/5 * * * * /home/hoborg/.local/scripts/duckdns.py >/dev/null 2>&1

Current Setup (Router-based)

Status: ✅ Migrated from script to router DynDNS

Changes made:

• ✅ Disabled cron job script (*/5 * * * * entry removed)
• ✅ Enabled router Dynamic DNS for ak-homelab.duckdns.org
• ⏳ Testing pending - Cannot force public IP change to verify

Router DynDNS Benefits:

• Immediate updates on IP change (vs 5-minute delay)
• Works when server is down
• Lower resource usage

Limitations:

• Likely IPv4-only (Sagemcom router limitation)
• Less control over update process

Testing

# Check current IP
curl -s https://ipinfo.io/ip

# Verify DNS resolution
nslookup ak-homelab.duckdns.org

# Check IPv6 (likely not updated by router)
nslookup -type=AAAA ak-homelab.duckdns.org

Testing will occur naturally when ISP changes public IP address.

VPN Setup with WireGuard

What is WireGuard?

WireGuard is a modern, lightweight VPN protocol that creates secure tunnels between devices. It encrypts all network traffic and routes it through a VPN server, making your internet connection private and secure.

Key benefits:

• Privacy: Hides your IP address and encrypts traffic
• Security: Protects against man-in-the-middle attacks on public WiFi
• Access: Bypass geo-restrictions and enables remote homelab access
• Performance: Much faster than OpenVPN with lower battery drain
• Simplicity: Easy to configure compared to other VPN protocols

When you need VPN:

• Accessing homelab remotely over internet
• Working from public WiFi frequently
• Need to bypass ISP restrictions
• Running public-facing services

Costs: WireGuard itself is free. Self-hosted VPN costs $5-20/month for VPS hosting.

Use cases:

• Access homelab services remotely (SSH, web interfaces, file shares)
• Secure connection on public WiFi
• Bypass ISP restrictions or geo-blocks

Performance: Much faster and lighter than OpenVPN, better battery life on mobile devices.

Server Configuration

Install WireGuard: pacman -S wireguard-tools

Generate keys:

wg genkey | tee server_private.key | wg pubkey > server_public.key

Server config /etc/wireguard/wg0.conf:

[Interface]
PrivateKey = <SERVER_PRIVATE_KEY>
Address = 10.0.0.1/24
ListenPort = 51820
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
PublicKey = <CLIENT_PUBLIC_KEY>
AllowedIPs = 10.0.0.2/32

Client Configuration

Generate client keys:

wg genkey | tee client_private.key | wg pubkey > client_public.key

Client config:

[Interface]
PrivateKey = <CLIENT_PRIVATE_KEY>
Address = 10.0.0.2/24
DNS = 1.1.1.1

[Peer]
PublicKey = <SERVER_PUBLIC_KEY>
Endpoint = ak-homelab.duckdns.org:51820
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 25

Enable VPN

sudo systemctl enable wg-quick@wg0
sudo systemctl start wg-quick@wg0

Firewall Configuration

UFW (Uncomplicated Firewall)

What it does: Controls what network traffic is allowed in/out of your server.

Key functions:

• Default deny: Blocks all incoming connections by default
• Port control: Open only specific ports you need (SSH, HTTP, etc.)
• Rate limiting: Prevent brute force attacks
• Application profiles: Pre-configured rules for common services

Why needed: Without firewall, all services are exposed to network attacks.

UFW Setup

# Install and enable UFW
pacman -S ufw
sudo ufw enable

# Default policies
sudo ufw default deny incoming
sudo ufw default allow outgoing

# SSH access
sudo ufw allow 2222/tcp

# WireGuard
sudo ufw allow 51820/udp

# HTTP/HTTPS for services
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

Advanced Rules

# Rate limiting for SSH
sudo ufw limit 2222/tcp

# Allow from specific networks
sudo ufw allow from 192.168.1.0/24 to any port 22

# Log denied connections
sudo ufw logging on

Network Security Best Practices

Port Management

• Change default ports: SSH (2222), WireGuard (51820)
• Close unused ports: Regular port scans with nmap
• Port forwarding: Only forward necessary ports

Access Control

• VPN-first approach: Access services through VPN tunnel
• IP whitelisting: Restrict access to known IPs when possible
• Rate limiting: Prevent brute force attacks

Monitoring

• Log analysis: Monitor /var/log/auth.log for SSH attempts
• Network monitoring: Use netstat/ss to check listening ports
• Intrusion detection: Use fail2ban for automated blocking

fail2ban - Intrusion Prevention

What is fail2ban?

What it does: Automatically blocks IP addresses that show malicious behavior.

Key functions:

• Log monitoring: Watches system logs for suspicious activity
• Pattern detection: Identifies failed login attempts, scanning, etc.
• Automatic blocking: Temporarily bans offending IP addresses
• Customizable rules: Configure what triggers a ban and for how long

Common protections:

• SSH brute force attempts
• Web server attacks (404 scanning, etc.)
• Email server abuse
• Custom application attacks

Example: After 5 failed SSH login attempts in 10 minutes, ban IP for 1 hour.

Why important: Reduces server load and prevents automated attacks from succeeding through persistence.

fail2ban Installation & Setup

# Install fail2ban
sudo pacman -S fail2ban

# Enable and start service
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

# Create local configuration
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Basic SSH Protection Configuration

Edit /etc/fail2ban/jail.local:

[sshd]
enabled = true
port = 2222
filter = sshd
logpath = /var/log/auth.log
maxretry = 5
bantime = 3600
findtime = 600

Configuration explained:

• maxretry = 5: Ban after 5 failed attempts
• bantime = 3600: Ban for 1 hour (3600 seconds)
• findtime = 600: 5 attempts within 10 minutes triggers ban
• port = 2222: Monitor custom SSH port

Restart and Monitor

# Restart fail2ban to apply changes
sudo systemctl restart fail2ban

# Check status
sudo fail2ban-client status
sudo fail2ban-client status sshd

# View banned IPs
sudo fail2ban-client get sshd banned

Geographic IP Blocking with geoip-shell

Overview

Status: ✅ Implemented - Whitelist mode protecting all interfaces

Tool: geoip-shell v0.7.5 Repository: https://github.com/friendly-bits/geoip-shell Purpose: Geographic IP blocking for enhanced security

Current Configuration

• Firewall Backend: iptables
• IP Lists Source: RIPE (Réseaux IP Européens)
• Network Interfaces: All interfaces protected
• LAN Detection: Automatic subnet detection enabled
• Mode: Whitelist (only specified countries allowed)
• IP Families: IPv4 and IPv6 supported
• Update Schedule: Daily at 4:18 AM
• Last Update: September 17, 2025 at 00:57:41

Whitelisted Countries

AL (Albania), AD (Andorra), AM (Armenia), AT (Austria), AZ (Azerbaijan)
BY (Belarus), BE (Belgium), BA (Bosnia and Herzegovina), BG (Bulgaria)
HR (Croatia), CY (Cyprus), CZ (Czech Republic), DK (Denmark)
EE (Estonia), FO (Faroe Islands), FI (Finland), FR (France)
GE (Georgia), DE (Germany), GI (Gibraltar), GR (Greece)
GG (Guernsey), HU (Hungary), IS (Iceland), IE (Ireland)
IM (Isle of Man), IT (Italy), JE (Jersey), KZ (Kazakhstan)
LV (Latvia), LI (Liechtenstein), LT (Lithuania), LU (Luxembourg)
MT (Malta), MD (Moldova), MC (Monaco), ME (Montenegro)
NL (Netherlands), MK (North Macedonia), NO (Norway), PL (Poland)
PT (Portugal), RO (Romania), RU (Russia), SM (San Marino)
RS (Serbia), SK (Slovakia), SI (Slovenia), ES (Spain)
SE (Sweden), CH (Switzerland), TR (Turkey), UA (Ukraine)
GB (United Kingdom), VA (Vatican City)

Network Exceptions (Always Allowed)

IPv4 Networks:

• 172.18.0.0/16 - Docker network
• 172.17.0.0/16 - Docker network
• 169.254.0.0/16 - Link-local addresses
• 192.168.0.0/24 - Local LAN
• 172.20.0.0/16 - Docker network
• 172.19.0.0/16 - Docker network

IPv6 Networks:

• fdaa:bbcc:ddee::/64 - Custom network
• fe80::/10 - Link-local addresses

Security Impact

1. Geographic Blocking: Blocks all traffic from countries not in whitelist
2. Comprehensive Coverage: Both IPv4 and IPv6 protection
3. Full Protocol Coverage: TCP and UDP traffic controlled
4. Network Awareness: Automatically detects and allows local networks
5. Persistence: Rules survive system reboots
6. Automatic Updates: IP lists updated daily

Integration with Existing Security

• Complements fail2ban: Provides geographic layer above intrusion detection
• Works with UFW: Uses iptables backend compatible with UFW
• Docker Compatible: Automatically allows Docker networks
• LAN Friendly: Preserves local network access

Monitoring

# Check geoip-shell status
geoip-shell status

# View logs
journalctl -u geoip-shell
tail -f /var/log/geoip-shell.log

Manual Setup Process

The tool was installed manually with interactive prompts rather than scripted installation due to security considerations. Key decisions made during setup:

1. Whitelist Mode: Chosen over blacklist for better control
2. European Focus: Primary whitelist consists of European countries
3. Network Exceptions: Docker and LAN networks automatically detected
4. Dual Stack: Both IPv4 and IPv6 protection enabled
5. Full Protocol Coverage: TCP and UDP both protected

Detailed Documentation

For complete setup details, see docs/geoip-blocking.md

Permanent Ban System for Repeat Offenders

Overview

Status: ✅ Implemented - Automated permanent banning of persistent attackers

Script: scripts/permanent-ban-repeat-offenders.sh Purpose: Automatically identify and permanently ban IPs that have been banned by fail2ban more than a threshold number of times

How It Works

Detection Logic

1. Log Analysis: Scans /var/log/fail2ban.log* for ban entries
2. IP Extraction: Extracts IP addresses from ban log entries
3. Frequency Counting: Counts how many times each IP has been banned
4. Threshold Check: Identifies IPs banned more than the threshold (4 times)

Permanent Banning Process

For each repeat offender:

1. Country Lookup: Uses whois to determine the country of origin
2. Banlist Update: Adds IP to /etc/fail2ban/permanent-banlist.conf
3. Firewall Rule: Creates permanent iptables DROP rule
4. Persistence: Saves iptables rules to /etc/iptables/iptables.rules
5. Service Reload: Reloads fail2ban to recognize the updated banlist

Configuration

Threshold Settings

THRESHOLD=4  # Ban after 4 fail2ban bans

File Locations

• Log File: /var/log/permanent-ban.log
• Banlist: /etc/fail2ban/permanent-banlist.conf
• Iptables Rules: /etc/iptables/iptables.rules

Cron Schedule

• Frequency: Daily at 2:00 AM (0 2 * * *)
• User: root
• Command: /home/hoborg/homelab/scripts/permanent-ban-repeat-offenders.sh

Security Benefits

Multi-Layer Protection

1. fail2ban: Temporary bans for suspicious activity
2. Permanent Bans: Long-term blocking of persistent attackers
3. Geographic Blocking: Country-level filtering via geoip-shell
4. Network-Level: iptables rules at the firewall level

Attack Prevention

• Brute Force: Blocks IPs that repeatedly attempt attacks
• Botnets: Prevents automated attacks from compromised hosts
• Persistence: Maintains bans across system reboots
• Resource Protection: Reduces server load from repeat offenders

Monitoring and Maintenance

Log Analysis

# View permanent ban activity
tail -f /var/log/permanent-ban.log

# Check current permanent bans
cat /etc/fail2ban/permanent-banlist.conf

# View iptables permanent rules
iptables -L | grep DROP

Manual Execution

# Run the script manually (requires root)
sudo /home/hoborg/homelab/scripts/permanent-ban-repeat-offenders.sh

Unban Procedure

To manually unban a permanently banned IP:

# Remove from banlist
sudo sed -i "/^192\.168\.1\.100/d" /etc/fail2ban/permanent-banlist.conf

# Remove iptables rule (find the rule number first)
sudo iptables -L --line-numbers | grep "192.168.1.100"
sudo iptables -D INPUT <rule_number>

# Save iptables rules
sudo iptables-save > /etc/iptables/iptables.rules

# Reload fail2ban
sudo systemctl reload fail2ban

Integration with Security Stack

Complementary Tools

• fail2ban: Provides temporary bans that feed into permanent ban detection
• geoip-shell: Geographic blocking at the network level
• UFW: Additional firewall layer
• SSH Hardening: Reduces initial attack surface

Workflow

Attack Attempt → fail2ban Detection → Temporary Ban → Repeat Offense → Permanent Ban → Geographic Block

Troubleshooting

Common Issues

• Script Not Running: Check cron job configuration
• Permission Errors: Ensure script is executable and paths are correct
• whois Failures: Some IPs may not return country information
• iptables-save Issues: Check if iptables-persistent is installed

Diagnostic Commands

# Check cron service
sudo systemctl status cron

# Test script manually
sudo bash -x /home/hoborg/homelab/scripts/permanent-ban-repeat-offenders.sh

# Verify iptables rules
sudo iptables -L -n | grep DROP

# Check fail2ban integration
sudo fail2ban-client status

Router Configuration

Port Forwarding

Forward these ports to your homelab server:

• SSH: External port → Internal 2222
• WireGuard: 51820 → 51820
• Web services: 80/443 → 80/443 (if needed)

Security Settings

• Disable WPS: Turn off WiFi Protected Setup
• Strong WiFi password: WPA3 with complex passphrase
• Guest network: Separate network for guests
• Firmware updates: Keep router firmware current

Network Planning

Dual Network Interface Issue (Critical)

Problem: Server has both ethernet and WiFi interfaces. When switching between connections, IP address changes from ethernet (192.168.0.22) to different WiFi IP, breaking SSH connections and port forwards.

Limitation: Most routers don't allow DHCP reservation of same IP for multiple MAC addresses.

Solutions:

Option 1: Static IP Configuration (Recommended) Configure both interfaces with same static IP:

# Check interface names
ip link show

# Configure ethernet interface
sudo systemctl edit --full systemd-networkd
# Create /etc/systemd/network/20-ethernet.network
[Match]
Name=enp*

[Network]
DHCP=no
Address=192.168.0.100/24
Gateway=192.168.0.1
DNS=192.168.0.1

# Create /etc/systemd/network/25-wifi.network
[Match]
Name=wlp*

[Network]
DHCP=no
Address=192.168.0.100/24
Gateway=192.168.0.1
DNS=192.168.0.1

Option 2: Hostname-based Access Use local hostname resolution instead of IP:

# Access via hostname (works for both interfaces)
ssh hoborg@ak-homelab.local
# or configure local DNS/mDNS

Option 3: Bridge Networking Create bridge combining both interfaces for automatic failover:

# Advanced: Bridge both interfaces
ip link add name br0 type bridge
ip link set enp3s0 master br0
ip link set wlp2s0 master br0

Current Setup:

• Router: 192.168.0.1
• Ethernet: 192.168.0.100 (static IP achieved)
• WiFi: Static IP needed (same .100)
• External: ak-homelab.duckdns.org ✅
• SSH: Port 2222 ✅

Network Boot Issues Troubleshooting

Problem 1: Network not available after reboot until GUI login

Initial diagnosis: Thought to be NetworkManager sleep behavior.

Actual root cause: System auto-suspend due to closed laptop lid (HandleLidSwitch=suspend).

Symptoms:

• SSH inaccessible after reboot
• Network comes up only when laptop lid is opened
• System logs show "Suspending..." followed by "Lid opened"

Solution Applied:

1. systemd-logind configuration: Disable lid switch handling for headless server operation
2. Static IP configuration: Eliminate DHCP negotiation delays
3. High connection priority: Ensure ethernet connects first

Configuration files created:

/etc/systemd/logind.conf.d/01-server-logind.conf:

[Login]
# Disable all power management triggers for headless server
HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore  
HandleLidSwitchDocked=ignore

# Prevent automatic suspend/hibernate
IdleAction=ignore

# Keep system running even when no users logged in
KillUserProcesses=no

NetworkManager commands:

nmcli connection modify "Wired connection 2" connection.autoconnect-priority 10
nmcli connection modify "Wired connection 2" ipv4.method manual ipv4.addresses 192.168.0.100/24 ipv4.gateway 192.168.0.1 ipv4.dns "84.2.44.8 84.2.46.8"

Deployment:

sudo cp config/systemd/01-server-logind.conf /etc/systemd/logind.conf.d/
sudo systemctl restart systemd-logind
sudo cp config/networkmanager/01-homelab.conf /etc/NetworkManager/conf.d/
sudo systemctl reload NetworkManager

Diagnostic Commands

Check system power state:

systemctl status systemd-logind  # Check for suspend/lid events
loginctl show-session           # Current power management settings

Check network connectivity:

systemctl status NetworkManager
nmcli device status
nmcli connection show "Wired connection 2"
ip addr show enp4s0

Monitor boot process:

journalctl -b -u NetworkManager    # Network startup logs
journalctl -b -u sshd             # SSH service logs  
journalctl -b -u systemd-logind   # Power management events

Result: ✅ RESOLVED - SSH accessible immediately on boot, lid closure no longer suspends system.

Network Interface Identification:

• enp3s0f0: First ethernet port (98:fa:9b:f1:06:d5)
• enp4s0: Second ethernet port (98:fa:9b:f1:06:d4) ← Use this one
• wlp1s0: WiFi interface (0c:dd:24:e6:0f:87)

Issue Solved: Dual ethernet ports caused MAC address confusion when cable was moved between ports. Stick to enp4s0 consistently.

IP Address Scheme

• Router: 192.168.0.1
• Homelab server: 192.168.0.100 (target static IP)
• Current ethernet: 192.168.0.22 (can migrate to .100)
• DHCP range: 192.168.0.10-99 (excluding static IPs)
• VPN subnet: 10.0.0.0/24

DNS Configuration

• Primary DNS: Router (192.168.1.1)
• Secondary DNS: 1.1.1.1, 8.8.8.8
• Local domain: homelab.local
• Dynamic DNS: ak-homelab.duckdns.org

Service Architecture

Internet → Router → Homelab Server
                 ↓
    ┌─────────────────────────────┐
    │  SSH (2222)                 │
    │  WireGuard VPN (51820)      │
    │  Web Services (80/443)      │
    │  Monitoring & Logging       │
    └─────────────────────────────┘

Backup and Recovery

Configuration Backups

• SSH keys: Store securely, separate from server
• WireGuard configs: Document peer configurations
• Firewall rules: Export UFW rules with ufw status numbered

Network Documentation

• IP mappings: Document static assignments
• Port forwards: List all forwarded ports and purposes
• Access credentials: Secure storage of passwords/keys