homelab/docs/troubleshooting/troubleshooting.md

# Bluetooth keeps disconnecting - reconnecting

Solution: Change config in /etc/bluetooth/main.conf
`ControllerMode = bredr`
Then `sudo systemctl restart bluetooth`

UPDATE: It's still not fixed :(
Trying `yay -S pipewire wireplumber`
TODO test it again

# Touchpad scroll direction (libinput)

To change touchpad scroll direction on Arch Linux using libinput driver:

## Investigation steps:
1. Check which driver is used: `pacman -Q | grep -E "(synaptics|libinput)"`
2. Verify libinput config exists: `ls /usr/share/X11/xorg.conf.d/ | grep libinput`

## Solution for libinput:
Create `/etc/X11/xorg.conf.d/30-touchpad.conf`:

```
Section "InputClass"
    Identifier "touchpad"
    Driver "libinput"
    MatchIsTouchpad "on"
    Option "NaturalScrolling" "true"
    Option "Tapping" "on"
    Option "TappingDrag" "on"
    Option "DisableWhileTyping" "on"
EndSection
```

Set `NaturalScrolling` to:
- `"true"` for macOS-style (natural) scrolling
- `"false"` for traditional scrolling

Restart X11 (log out/in) or reboot to apply changes.

## Alternative: Synaptics driver (legacy)

If using the older synaptics driver instead of libinput:

### Investigation steps:
1. Check for synaptics: `pacman -Q xf86-input-synaptics`
2. Look for config: `ls /usr/share/X11/xorg.conf.d/ | grep synaptics`

### Solution:
Create `/etc/X11/xorg.conf.d/70-synaptics.conf`:

```
Section "InputClass"
    Identifier "touchpad"
    Driver "synaptics"
    MatchIsTouchpad "on"
    MatchDevicePath "/dev/input/event*"
    Option "VertScrollDelta" "-111"
    Option "HorizScrollDelta" "-111"
    Option "TapButton1" "1"
    Option "TapButton2" "3"
    Option "TapButton3" "2"
    Option "PalmDetect" "1"
    Option "SHMConfig" "on"
EndSection
```

Synaptics scroll direction options:
- `VertScrollDelta` and `HorizScrollDelta`:
  - Positive values (e.g., `"111"`) for traditional scrolling
  - Negative values (e.g., `"-111"`) for natural/reversed scrolling

**Note:** libinput is the modern standard. Consider switching from synaptics to libinput for better support and features.

# Theme switching issues

## Cross-application theme synchronization

**Issue:** Need to synchronize theme (light/dark) across tmux, nvim, and other terminal applications.

**Solution:** File-based theme management system using `~/.vim_theme`:

### Setup:

1. **Create theme switcher script** (`~/.config/tmux/themeswitch.sh`):
```bash
#!/bin/bash
if [ -f ~/.vim_theme ] && [ "$(cat ~/.vim_theme)" = "light" ]; then
    tmux set -g @catppuccin_flavor "latte"
else
    tmux set -g @catppuccin_flavor "mocha"
fi
```

2. **Add to tmux config** (`~/.config/tmux/tmux.conf`):
```bash
# Dynamic theme switching based on ~/.vim_theme file
run 'bash ~/.config/tmux/themeswitch.sh'
```

3. **Add to shell config** (`~/.zshrc`):
```bash
# Export THEME environment variable for nvim and other apps
if [ -f ~/.vim_theme ]; then
    export THEME=$(cat ~/.vim_theme)
else
    export THEME="dark"  # default
fi
```

### Usage:
```bash
# Switch to light theme
echo "light" > ~/.vim_theme

# Switch to dark theme
echo "dark" > ~/.vim_theme
```

### Benefits:
- ✅ **Single source of truth:** `~/.vim_theme` file controls all applications
- ✅ **Automatic propagation:** New terminals inherit theme via shell config
- ✅ **No environment variable issues:** File-based approach avoids tmux env var propagation problems
- ✅ **Cross-application support:** nvim reads `$THEME`, tmux uses catppuccin flavors

### Limitations:
- **Tmux live reload:** Changes require tmux config reload or session restart
- **Workaround:** Use tmux-resurrect to quickly restore sessions after restart

## Legacy: Tmux and terminal not updating after theme switch

**Issue:** After running the theme switcher script, tmux sessions and existing terminals don't reflect the new theme until restarted.

**Temporary workaround:**
- Restart tmux sessions: `tmux kill-server && tmux`
- Open new terminal windows

**Status:** ✅ **Solved** - Use file-based theme management system above

## Tmux window names showing hostname instead of command

**Issue:** Tmux windows show "homelab" (hostname) for inactive tabs but correct command names for active tabs.

**Root cause:** Catppuccin tmux theme with `@catppuccin_window_tabs_enabled on` uses different text formatting for active vs inactive windows.

**Solution:** Disable catppuccin window tabs:
```
set -g @catppuccin_window_tabs_enabled off
```

**Alternative:** Configure explicit window text for both states:
```
set -g @catppuccin_window_default_text "#W"
set -g @catppuccin_window_current_text "#W"
```

Also ensure automatic renaming is enabled:
```
setw -g automatic-rename on
setw -g allow-rename on
```

# Font and Unicode Display Issues

## Missing emoji and unicode symbols

**Issue:** Emojis show as boxes or missing characters, unicode symbols don't display properly.

**Solution:** Install comprehensive unicode font packages:
```bash
sudo pacman -S noto-fonts-emoji noto-fonts-extra
fc-cache -f
```

## Nerd Font icons not displaying

**Issue:** Developer icons (programming languages, git symbols, file types) show as blank spaces or boxes.

**Root cause:** Terminal emulator not configured to use Nerd Font as primary font.

**Solution:**
1. Install Nerd Fonts:
```bash
sudo pacman -S ttf-iosevkaterm-nerd ttf-jetbrains-mono-nerd
fc-cache -f
```

2. Configure terminal to use Nerd Font as primary font
3. For wezterm, ensure config includes:
```lua
config.font = wezterm.font_with_fallback {
    'IosevkaTerm Nerd Font',
    'JetBrainsMono Nerd Font Mono',
    'Noto Color Emoji'
}
```

**Testing:** Use printf with direct codepoints:
```bash
printf "Icons: \\ue702 \\uf121 \\uf015 \\uf07b\\n"
```

## Ancient/exotic script support

**Comprehensive coverage achieved with:**
- `noto-fonts` (base unicode)
- `noto-fonts-cjk` (Chinese/Japanese/Korean)
- `noto-fonts-emoji` (color emoji)
- `noto-fonts-extra` (additional scripts)

Successfully displays: Egyptian hieroglyphs, Cuneiform, Nordic runes, Hungarian rovás, Arabic, Chinese, Japanese, Korean, Thai, Hindi, Hebrew, Greek, Tamil.

# Cannot tile windows by drag and dropping

Keyboard workaround: Go to  Settings > Window Manager > Keyboard, set up tiling shortcuts (set to Super+arrow keys)

# Additional Known Issues (TODO Items)

## Tmux battery indicator missing until config reload

**Issue:** Battery indicator doesn't appear in tmux status line immediately after starting tmux.

**Temporary workaround:** Reload tmux config with `Prefix + r` or restart tmux session.

**Status:** Investigation needed

## TTY fallbacks needed

**Issue:** When not in X11/graphical mode, nvim and tmux need proper fallback configurations.

**Status:** Completed

**Solutions implemented:**
- ✅ **nvim:** TTY detection and color scheme fallback configured
- ✅ **Font:** Selected ter-124b (12x24 bold) for good readability
- ✅ **Keyboard:** Colemak layout with caps lock remapped to backspace
- ✅ **Caps lock fix:** Uses systemd service with `setkeycodes 3a 14`

**Configuration files:**
- `/etc/systemd/system/caps-backspace.service` - Permanent caps lock remapping
- TTY font testing script: `~/.local/scripts/test-fonts.sh`

## TTY Caps Lock Not Working as Backspace

**Issue:** With colemak keymap loaded, caps lock acts like Control instead of backspace in TTY.

**Root cause:** Colemak keymap maps caps lock to Control, which conflicts with tmux navigation keys.

**Solution:** Use `setkeycodes` to remap at scancode level:
```bash
sudo setkeycodes 3a 14  # Map caps lock scancode to backspace keycode
```

**Permanent fix:** Systemd service created at `/etc/systemd/system/caps-backspace.service`

## Laptop sleeps when lid is closed

Solution:

sudo nvim /etc/systemd/logind.conf

Uncomment and change these lines:

```
HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore
HandleLidSwitchDocked=ignore
```

Then restart service:
`sudo systemctl restart systemd-logind`

## IP addresses keep changing

Due to 3 interfaces:
* Wifi
* Ethernet left port (ThinkPad adapter) - Wired Connection 1
* Ethernet right port (regular ethernet cable) - Wired Connection 2

### Proposed solution

* Stick with wired2

● Configure static IP for "Wired connection 2" (enp4s0):

  * Set static IP to 192.168.0.100
  sudo nmcli connection modify "Wired connection 2" \
      ipv4.method manual \
      ipv4.addresses 192.168.0.100/24 \
      ipv4.gateway 192.168.0.1 \
      ipv4.dns 192.168.0.1

  * Apply the changes
  sudo nmcli connection up "Wired connection 2"

  Then configure WiFi with the same static IP:

  * First connect to your WiFi if not already
  sudo nmcli connection up "Telekom-4b28df-2.4GHz"

  * Set same static IP for WiFi
  sudo nmcli connection modify "Telekom-4b28df-2.4GHz" \
      ipv4.method manual \
      ipv4.addresses 192.168.0.100/24 \
      ipv4.gateway 192.168.0.1 \
      ipv4.dns 192.168.0.1

  Verify the configuration:
  nmcli connection show "Wired connection 2" | grep ipv4
  ip addr show enp4s0

  This way both your ethernet (enp4s0) and WiFi will use 192.168.0.100, solving your dual interface IP issue.
  Ready to run these commands?

# Homelab Service Issues

## Landing Page Returns 404 Not Found

**Issue:** Landing page at `https://ak-homelab.duckdns.org/` returns 404 Not Found despite nginx running and other services working.

**Symptoms:**
- Landing page returns 404 Not Found
- Nginx logs show attempts to access `/etc/nginx/html/index.html` instead of `/var/www/homelab/index.html`
- Custom 404 pages work but main index doesn't load
- Services like `/gitea/`, `/files/`, `/media/` work correctly

**Root Cause:**
1. Default server block in `/etc/nginx/nginx.conf` conflicted with homelab configuration
2. Missing `default_server` directive in homelab server blocks
3. Incorrect location block (`location = /` vs `location /`)
4. Nginx not properly reloading configuration changes

**Resolution Steps:**
```bash
# 1. Comment out default server block in nginx.conf
sudo nano /etc/nginx/nginx.conf
# Comment out the entire server { listen 80; server_name localhost; ... } block

# 2. Add default_server directives to homelab config
sudo nano /etc/nginx/sites-available/homelab
# Change to: listen 80 default_server; and listen 443 ssl default_server;
# Add server_name ak-homelab.duckdns.org _; (underscore catches all)

# 3. Fix location block for root path
# Change from: location = / { ... }
# Change to:   location / { root /var/www/homelab; index index.html; try_files $uri $uri/ =404; }

# 4. Add custom error pages to prevent fallback
# Add: error_page 404 /404.html;
#      location = /404.html { root /var/www/homelab; internal; }

# 5. Test and restart (reload may not be sufficient)
sudo nginx -t
sudo systemctl restart nginx  # Use restart, not reload

# 6. Verify file permissions
sudo ls -la /var/www/homelab/index.html
# Should be readable by nginx user (http)
```

**Verification Commands:**
```bash
# Test HTTP redirect
curl -I http://192.168.0.100/

# Test HTTPS access
curl -I https://ak-homelab.duckdns.org/ --insecure

# Check active nginx configuration
sudo nginx -T | grep -A10 "server_name ak-homelab"

# Monitor real-time nginx logs
sudo journalctl -u nginx -f
```

**Final Working Configuration:**
- HTTP redirects properly to HTTPS (301 redirect)
- HTTPS returns 200 OK with full landing page HTML
- Security headers present (X-Frame-Options, X-Content-Type-Options, X-XSS-Protection)
- Custom error handling prevents fallback to default nginx errors

## SSL Certificate Breaks After Nginx Updates

**Issue:** HTTPS stops working after deploying nginx configuration changes from the repo.

**Root cause:** Certbot modifies the deployed nginx configuration directly (adding SSL blocks), but the repo only contains the base HTTP configuration. When deploying from repo, it overwrites certbot's SSL additions.

**Solution:**
```bash
# 1. Re-run certbot to restore SSL configuration
sudo certbot --nginx -d ak-homelab.duckdns.org --non-interactive

# 2. Copy complete config (with SSL blocks) back to repo
sudo cp /etc/nginx/sites-available/homelab config/nginx/homelab.conf

# 3. Commit updated config to prevent future issues
git add config/nginx/homelab.conf
git commit -m "Update nginx config with SSL configuration from certbot"
```

**Prevention:** Always copy deployed configurations back to repo after certbot runs to preserve SSL blocks.

## WebDAV Upload Error 301 (Moved Permanently)

**Issue:** WebDAV clients (X-plore, rclone) get HTTP 301 redirect errors when uploading files, even though browsing works fine.

**Root cause:** Nginx reverse proxy missing WebDAV-specific headers needed for upload operations.

**Solution:** Ensure nginx configuration includes WebDAV headers:
```nginx
location /files/ {
    proxy_pass http://127.0.0.1:8082/files/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    
    # WebDAV specific headers (REQUIRED for uploads)
    proxy_set_header Depth $http_depth;
    proxy_set_header Destination $http_destination;
    proxy_set_header Overwrite $http_overwrite;
    proxy_set_header If $http_if;
    
    proxy_request_buffering off;
}
```

**WebDAV Client Settings (X-plore):**
- URL: `https://ak-homelab.duckdns.org/files/` 
- Username: `hoborg`
- Password: `AdminPass2024!`
- Port: 443 (HTTPS)

## Media Files Not Syncing Between Copyparty and Jellyfin

**Issue:** Files uploaded to Copyparty don't appear in Jellyfin media library.

**Root cause:** Volume mount mismatch or Jellyfin not scanning new files.

**Troubleshooting:**
```bash
# 1. Verify volume mounts match
# Copyparty config: /home/hoborg/private -> /private
# Jellyfin config: /home/hoborg/private:/media/private:ro

# 2. Check file permissions
ls -la /home/hoborg/private/
# Files should be readable by user 1000:1000

# 3. Force Jellyfin library rescan
docker exec jellyfin curl -X POST "http://localhost:8096/Library/Refresh"

# 4. Check Jellyfin logs
docker-compose -f /opt/docker/jellyfin/docker-compose.yml logs jellyfin
```

## Service Deployment Best Practices

**To avoid common issues:**

1. **Always test nginx before reload:**
```bash
sudo nginx -t && sudo systemctl reload nginx
```

2. **Backup configs before changes:**
```bash
sudo cp /etc/nginx/sites-available/homelab /etc/nginx/sites-available/homelab.backup
```

3. **Use temporary scripts for multiple sudo commands:**
```bash
# Create script with all commands, then run once with sudo
echo "command1; command2; command3" > /tmp/deploy.sh
chmod +x /tmp/deploy.sh
sudo -A /tmp/deploy.sh
```

4. **Monitor service logs after changes:**
```bash
sudo systemctl status nginx copyparty
docker-compose -f /opt/docker/jellyfin/docker-compose.yml logs --tail 50
```

## Configuration Debugging Commands

### Nginx Configuration Analysis
```bash
# Show all active server blocks
sudo nginx -T | grep -A50 "server {"

# Find all nginx config files
sudo find /etc/nginx -name "*.conf" -exec grep -l "server_name\|listen.*80\|listen.*443" {} \;

# Check which server block is handling requests
sudo nginx -T | grep -A20 "location = /"

# Verify sites-enabled vs sites-available
sudo ls -la /etc/nginx/sites-enabled/
sudo ls -la /etc/nginx/sites-available/

# Test nginx config without reloading
sudo nginx -t
```

### File Permissions and Access
```bash
# Check file permissions for web content
sudo ls -la /var/www/homelab/
sudo -u http ls -la /var/www/homelab/  # Test as nginx user

# Check SSL certificate access
sudo ls -la /etc/letsencrypt/live/ak-homelab.duckdns.org/

# Verify configuration deployment
diff /home/hoborg/homelab/config/nginx/homelab.conf /etc/nginx/sites-available/homelab
```

### Real-time Debugging
```bash
# Monitor nginx logs in real-time
sudo journalctl -u nginx -f

# Monitor nginx logs since specific time
sudo journalctl -u nginx --since "10 minutes ago"

# Test with verbose curl output
curl -v https://ak-homelab.duckdns.org/ --insecure

# Check nginx worker processes
ps aux | grep nginx
```

### Service Status Verification
```bash
# Check all homelab services at once
sudo systemctl status nginx copyparty
docker-compose -f /opt/docker/gitea/docker-compose.yml ps
docker-compose -f /opt/docker/jellyfin/docker-compose.yml ps

# Quick connectivity test for all services
curl -I https://ak-homelab.duckdns.org/        # Landing page
curl -I https://ak-homelab.duckdns.org/gitea/  # Gitea
curl -I https://ak-homelab.duckdns.org/files/  # Copyparty
curl -I https://ak-homelab.duckdns.org/media/  # Jellyfin
```