homelab/docs/troubleshooting/troubleshooting.md

Bluetooth keeps disconnecting - reconnecting

Solution: Change config in /etc/bluetooth/main.conf ControllerMode = bredr Then sudo systemctl restart bluetooth

UPDATE: It's still not fixed :( Trying yay -S pipewire wireplumber TODO test it again

Touchpad scroll direction (libinput)

To change touchpad scroll direction on Arch Linux using libinput driver:

Investigation steps:

1. Check which driver is used: pacman -Q | grep -E "(synaptics|libinput)"
2. Verify libinput config exists: ls /usr/share/X11/xorg.conf.d/ | grep libinput

Solution for libinput:

Create /etc/X11/xorg.conf.d/30-touchpad.conf:

Section "InputClass"
    Identifier "touchpad"
    Driver "libinput"
    MatchIsTouchpad "on"
    Option "NaturalScrolling" "true"
    Option "Tapping" "on"
    Option "TappingDrag" "on"
    Option "DisableWhileTyping" "on"
EndSection

Set NaturalScrolling to:

• "true" for macOS-style (natural) scrolling
• "false" for traditional scrolling

Restart X11 (log out/in) or reboot to apply changes.

Alternative: Synaptics driver (legacy)

If using the older synaptics driver instead of libinput:

Investigation steps:

1. Check for synaptics: pacman -Q xf86-input-synaptics
2. Look for config: ls /usr/share/X11/xorg.conf.d/ | grep synaptics

Solution:

Create /etc/X11/xorg.conf.d/70-synaptics.conf:

Section "InputClass"
    Identifier "touchpad"
    Driver "synaptics"
    MatchIsTouchpad "on"
    MatchDevicePath "/dev/input/event*"
    Option "VertScrollDelta" "-111"
    Option "HorizScrollDelta" "-111"
    Option "TapButton1" "1"
    Option "TapButton2" "3"
    Option "TapButton3" "2"
    Option "PalmDetect" "1"
    Option "SHMConfig" "on"
EndSection

Synaptics scroll direction options:

• VertScrollDelta and HorizScrollDelta:
  • Positive values (e.g., "111") for traditional scrolling
  • Negative values (e.g., "-111") for natural/reversed scrolling

Note: libinput is the modern standard. Consider switching from synaptics to libinput for better support and features.

Theme switching issues

Cross-application theme synchronization

Issue: Need to synchronize theme (light/dark) across tmux, nvim, and other terminal applications.

Solution: File-based theme management system using ~/.vim_theme:

Setup:

1. Create theme switcher script (~/.config/tmux/themeswitch.sh):

#!/bin/bash
if [ -f ~/.vim_theme ] && [ "$(cat ~/.vim_theme)" = "light" ]; then
    tmux set -g @catppuccin_flavor "latte"
else
    tmux set -g @catppuccin_flavor "mocha"
fi

2. Add to tmux config (~/.config/tmux/tmux.conf):

# Dynamic theme switching based on ~/.vim_theme file
run 'bash ~/.config/tmux/themeswitch.sh'

3. Add to shell config (~/.zshrc):

# Export THEME environment variable for nvim and other apps
if [ -f ~/.vim_theme ]; then
    export THEME=$(cat ~/.vim_theme)
else
    export THEME="dark"  # default
fi

Usage:

# Switch to light theme
echo "light" > ~/.vim_theme

# Switch to dark theme
echo "dark" > ~/.vim_theme

Benefits:

• ✅ Single source of truth: ~/.vim_theme file controls all applications
• ✅ Automatic propagation: New terminals inherit theme via shell config
• ✅ No environment variable issues: File-based approach avoids tmux env var propagation problems
• ✅ Cross-application support: nvim reads $THEME, tmux uses catppuccin flavors

Limitations:

• Tmux live reload: Changes require tmux config reload or session restart
• Workaround: Use tmux-resurrect to quickly restore sessions after restart

Legacy: Tmux and terminal not updating after theme switch

Issue: After running the theme switcher script, tmux sessions and existing terminals don't reflect the new theme until restarted.

Temporary workaround:

• Restart tmux sessions: tmux kill-server && tmux
• Open new terminal windows

Status: ✅ Solved - Use file-based theme management system above

Tmux window names showing hostname instead of command

Issue: Tmux windows show "homelab" (hostname) for inactive tabs but correct command names for active tabs.

Root cause: Catppuccin tmux theme with @catppuccin_window_tabs_enabled on uses different text formatting for active vs inactive windows.

Solution: Disable catppuccin window tabs:

set -g @catppuccin_window_tabs_enabled off

Alternative: Configure explicit window text for both states:

set -g @catppuccin_window_default_text "#W"
set -g @catppuccin_window_current_text "#W"

Also ensure automatic renaming is enabled:

setw -g automatic-rename on
setw -g allow-rename on

Font and Unicode Display Issues

Missing emoji and unicode symbols

Issue: Emojis show as boxes or missing characters, unicode symbols don't display properly.

Solution: Install comprehensive unicode font packages:

sudo pacman -S noto-fonts-emoji noto-fonts-extra
fc-cache -f

Nerd Font icons not displaying

Issue: Developer icons (programming languages, git symbols, file types) show as blank spaces or boxes.

Root cause: Terminal emulator not configured to use Nerd Font as primary font.

Solution:

1. Install Nerd Fonts:

sudo pacman -S ttf-iosevkaterm-nerd ttf-jetbrains-mono-nerd
fc-cache -f

2. Configure terminal to use Nerd Font as primary font
3. For wezterm, ensure config includes:

config.font = wezterm.font_with_fallback {
    'IosevkaTerm Nerd Font',
    'JetBrainsMono Nerd Font Mono',
    'Noto Color Emoji'
}

Testing: Use printf with direct codepoints:

printf "Icons: \\ue702 \\uf121 \\uf015 \\uf07b\\n"

Ancient/exotic script support

Comprehensive coverage achieved with:

• noto-fonts (base unicode)
• noto-fonts-cjk (Chinese/Japanese/Korean)
• noto-fonts-emoji (color emoji)
• noto-fonts-extra (additional scripts)

Successfully displays: Egyptian hieroglyphs, Cuneiform, Nordic runes, Hungarian rovás, Arabic, Chinese, Japanese, Korean, Thai, Hindi, Hebrew, Greek, Tamil.

Cannot tile windows by drag and dropping

Keyboard workaround: Go to Settings > Window Manager > Keyboard, set up tiling shortcuts (set to Super+arrow keys)

Additional Known Issues (TODO Items)

Tmux battery indicator missing until config reload

Issue: Battery indicator doesn't appear in tmux status line immediately after starting tmux.

Temporary workaround: Reload tmux config with Prefix + r or restart tmux session.

Status: Investigation needed

TTY fallbacks needed

Issue: When not in X11/graphical mode, nvim and tmux need proper fallback configurations.

Status: Completed

Solutions implemented:

• ✅ nvim: TTY detection and color scheme fallback configured
• ✅ Font: Selected ter-124b (12x24 bold) for good readability
• ✅ Keyboard: Colemak layout with caps lock remapped to backspace
• ✅ Caps lock fix: Uses systemd service with setkeycodes 3a 14

Configuration files:

• /etc/systemd/system/caps-backspace.service - Permanent caps lock remapping
• TTY font testing script: ~/.local/scripts/test-fonts.sh

TTY Caps Lock Not Working as Backspace

Issue: With colemak keymap loaded, caps lock acts like Control instead of backspace in TTY.

Root cause: Colemak keymap maps caps lock to Control, which conflicts with tmux navigation keys.

Solution: Use setkeycodes to remap at scancode level:

sudo setkeycodes 3a 14  # Map caps lock scancode to backspace keycode

Permanent fix: Systemd service created at /etc/systemd/system/caps-backspace.service

Laptop sleeps when lid is closed

Solution:

sudo nvim /etc/systemd/logind.conf

Uncomment and change these lines:

HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore
HandleLidSwitchDocked=ignore

Then restart service: sudo systemctl restart systemd-logind

IP addresses keep changing

Due to 3 interfaces:

• Wifi
• Ethernet left port (ThinkPad adapter) - Wired Connection 1
• Ethernet right port (regular ethernet cable) - Wired Connection 2

Proposed solution

• Stick with wired2

● Configure static IP for "Wired connection 2" (enp4s0):

• Set static IP to 192.168.0.100 sudo nmcli connection modify "Wired connection 2"
  ipv4.method manual
  ipv4.addresses 192.168.0.100/24
  ipv4.gateway 192.168.0.1
  ipv4.dns 192.168.0.1

• Apply the changes sudo nmcli connection up "Wired connection 2"

Then configure WiFi with the same static IP:

• First connect to your WiFi if not already sudo nmcli connection up "Telekom-4b28df-2.4GHz"

• Set same static IP for WiFi sudo nmcli connection modify "Telekom-4b28df-2.4GHz"
  ipv4.method manual
  ipv4.addresses 192.168.0.100/24
  ipv4.gateway 192.168.0.1
  ipv4.dns 192.168.0.1

Verify the configuration: nmcli connection show "Wired connection 2" | grep ipv4 ip addr show enp4s0

This way both your ethernet (enp4s0) and WiFi will use 192.168.0.100, solving your dual interface IP issue. Ready to run these commands?

Homelab Service Issues

Landing Page Returns 404 Not Found

Issue: Landing page at https://ak-homelab.duckdns.org/ returns 404 Not Found despite nginx running and other services working.

Symptoms:

• Landing page returns 404 Not Found
• Nginx logs show attempts to access /etc/nginx/html/index.html instead of /var/www/homelab/index.html
• Custom 404 pages work but main index doesn't load
• Services like /gitea/, /files/, /media/ work correctly

Root Cause:

1. Default server block in /etc/nginx/nginx.conf conflicted with homelab configuration
2. Missing default_server directive in homelab server blocks
3. Incorrect location block (location = / vs location /)
4. Nginx not properly reloading configuration changes

Resolution Steps:

# 1. Comment out default server block in nginx.conf
sudo nano /etc/nginx/nginx.conf
# Comment out the entire server { listen 80; server_name localhost; ... } block

# 2. Add default_server directives to homelab config
sudo nano /etc/nginx/sites-available/homelab
# Change to: listen 80 default_server; and listen 443 ssl default_server;
# Add server_name ak-homelab.duckdns.org _; (underscore catches all)

# 3. Fix location block for root path
# Change from: location = / { ... }
# Change to:   location / { root /var/www/homelab; index index.html; try_files $uri $uri/ =404; }

# 4. Add custom error pages to prevent fallback
# Add: error_page 404 /404.html;
#      location = /404.html { root /var/www/homelab; internal; }

# 5. Test and restart (reload may not be sufficient)
sudo nginx -t
sudo systemctl restart nginx  # Use restart, not reload

# 6. Verify file permissions
sudo ls -la /var/www/homelab/index.html
# Should be readable by nginx user (http)

Verification Commands:

# Test HTTP redirect
curl -I http://192.168.0.100/

# Test HTTPS access
curl -I https://ak-homelab.duckdns.org/ --insecure

# Check active nginx configuration
sudo nginx -T | grep -A10 "server_name ak-homelab"

# Monitor real-time nginx logs
sudo journalctl -u nginx -f

Final Working Configuration:

• HTTP redirects properly to HTTPS (301 redirect)
• HTTPS returns 200 OK with full landing page HTML
• Security headers present (X-Frame-Options, X-Content-Type-Options, X-XSS-Protection)
• Custom error handling prevents fallback to default nginx errors

SSL Certificate Breaks After Nginx Updates

Issue: HTTPS stops working after deploying nginx configuration changes from the repo.

Root cause: Certbot modifies the deployed nginx configuration directly (adding SSL blocks), but the repo only contains the base HTTP configuration. When deploying from repo, it overwrites certbot's SSL additions.

Solution:

# 1. Re-run certbot to restore SSL configuration
sudo certbot --nginx -d ak-homelab.duckdns.org --non-interactive

# 2. Copy complete config (with SSL blocks) back to repo
sudo cp /etc/nginx/sites-available/homelab config/nginx/homelab.conf

# 3. Commit updated config to prevent future issues
git add config/nginx/homelab.conf
git commit -m "Update nginx config with SSL configuration from certbot"

Prevention: Always copy deployed configurations back to repo after certbot runs to preserve SSL blocks.

WebDAV Upload Error 301 (Moved Permanently)

Issue: WebDAV clients (X-plore, rclone) get HTTP 301 redirect errors when uploading files, even though browsing works fine.

Root cause: Nginx reverse proxy missing WebDAV-specific headers needed for upload operations.

Solution: Ensure nginx configuration includes WebDAV headers:

location /files/ {
    proxy_pass http://127.0.0.1:8082/files/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    
    # WebDAV specific headers (REQUIRED for uploads)
    proxy_set_header Depth $http_depth;
    proxy_set_header Destination $http_destination;
    proxy_set_header Overwrite $http_overwrite;
    proxy_set_header If $http_if;
    
    proxy_request_buffering off;
}

WebDAV Client Settings (X-plore):

• URL: https://ak-homelab.duckdns.org/files/
• Username: hoborg
• Password: AdminPass2024!
• Port: 443 (HTTPS)

Media Files Not Syncing Between Copyparty and Jellyfin

Issue: Files uploaded to Copyparty don't appear in Jellyfin media library.

Root cause: Volume mount mismatch or Jellyfin not scanning new files.

Troubleshooting:

# 1. Verify volume mounts match
# Copyparty config: /home/hoborg/private -> /private
# Jellyfin config: /home/hoborg/private:/media/private:ro

# 2. Check file permissions
ls -la /home/hoborg/private/
# Files should be readable by user 1000:1000

# 3. Force Jellyfin library rescan
docker exec jellyfin curl -X POST "http://localhost:8096/Library/Refresh"

# 4. Check Jellyfin logs
docker-compose -f /opt/docker/jellyfin/docker-compose.yml logs jellyfin

Service Deployment Best Practices

To avoid common issues:

1. Always test nginx before reload:

sudo nginx -t && sudo systemctl reload nginx

2. Backup configs before changes:

sudo cp /etc/nginx/sites-available/homelab /etc/nginx/sites-available/homelab.backup

3. Use temporary scripts for multiple sudo commands:

# Create script with all commands, then run once with sudo
echo "command1; command2; command3" > /tmp/deploy.sh
chmod +x /tmp/deploy.sh
sudo -A /tmp/deploy.sh

4. Monitor service logs after changes:

sudo systemctl status nginx copyparty
docker-compose -f /opt/docker/jellyfin/docker-compose.yml logs --tail 50

Configuration Debugging Commands

Nginx Configuration Analysis

# Show all active server blocks
sudo nginx -T | grep -A50 "server {"

# Find all nginx config files
sudo find /etc/nginx -name "*.conf" -exec grep -l "server_name\|listen.*80\|listen.*443" {} \;

# Check which server block is handling requests
sudo nginx -T | grep -A20 "location = /"

# Verify sites-enabled vs sites-available
sudo ls -la /etc/nginx/sites-enabled/
sudo ls -la /etc/nginx/sites-available/

# Test nginx config without reloading
sudo nginx -t

File Permissions and Access

# Check file permissions for web content
sudo ls -la /var/www/homelab/
sudo -u http ls -la /var/www/homelab/  # Test as nginx user

# Check SSL certificate access
sudo ls -la /etc/letsencrypt/live/ak-homelab.duckdns.org/

# Verify configuration deployment
diff /home/hoborg/homelab/config/nginx/homelab.conf /etc/nginx/sites-available/homelab

Real-time Debugging

# Monitor nginx logs in real-time
sudo journalctl -u nginx -f

# Monitor nginx logs since specific time
sudo journalctl -u nginx --since "10 minutes ago"

# Test with verbose curl output
curl -v https://ak-homelab.duckdns.org/ --insecure

# Check nginx worker processes
ps aux | grep nginx

Service Status Verification

# Check all homelab services at once
sudo systemctl status nginx copyparty
docker-compose -f /opt/docker/gitea/docker-compose.yml ps
docker-compose -f /opt/docker/jellyfin/docker-compose.yml ps

# Quick connectivity test for all services
curl -I https://ak-homelab.duckdns.org/        # Landing page
curl -I https://ak-homelab.duckdns.org/gitea/  # Gitea
curl -I https://ak-homelab.duckdns.org/files/  # Copyparty
curl -I https://ak-homelab.duckdns.org/media/  # Jellyfin