homelab/docs/ssh-intrusion-monitoring.md

# SSH Intrusion Monitoring Guide

## Overview

This guide covers comprehensive SSH intrusion detection and monitoring for the homelab environment, including both honeypot analysis and real SSH service monitoring.

## Log Files and Locations

### Primary Log Files
- **Honeypot Logs**: `/var/log/honeypot.log` - Attack attempts on port 22
- **SSH Authentication**: `/var/log/auth.log` - Real SSH attempts on port 2222
- **System Security**: `/var/log/secure` or `/var/log/security.log` - General security events
- **Service Logs**: `journalctl -u sshd.service` - SSH daemon logs

### Service Status Monitoring
```bash
# Check honeypot service
systemctl status ssh-honeypot.service

# Check real SSH service  
systemctl status sshd.service

# Monitor both services
systemctl status ssh-honeypot.service sshd.service
```

## Honeypot Monitoring

### Real-time Monitoring
```bash
# Live honeypot attack monitoring
tail -f /var/log/honeypot.log

# Monitor with timestamps
tail -f /var/log/honeypot.log | while read line; do echo "$(date '+%H:%M:%S') $line"; done
```

### Attack Analysis
```bash
# Count total attack attempts
grep -c "honeypot connection" /var/log/honeypot.log

# Show unique attacking IP addresses with attempt counts
grep "honeypot connection" /var/log/honeypot.log | \
  awk '{print $NF}' | sort | uniq -c | sort -nr

# Attacks by hour of day
grep "honeypot connection" /var/log/honeypot.log | \
  awk '{print $4}' | cut -d: -f1 | sort | uniq -c | sort -k2 -n

# Recent attacks (last 24 hours)
grep "$(date '+%a %b %d')" /var/log/honeypot.log

# Attacks from specific IP
grep "192.168.1.100" /var/log/honeypot.log
```

### Geographic Analysis
```bash
# Get country information for attacking IPs (requires geoip)
grep "honeypot connection" /var/log/honeypot.log | \
  awk '{print $NF}' | sort -u | \
  while read ip; do
    echo -n "$ip: "
    geoiplookup "$ip" 2>/dev/null | head -1 | cut -d: -f2
  done
```

## Real SSH Monitoring

### Authentication Monitoring
```bash
# Monitor real SSH authentication attempts
tail -f /var/log/auth.log | grep sshd

# Failed password attempts
grep "Failed password" /var/log/auth.log | tail -10

# Successful logins
grep "Accepted password\|Accepted publickey" /var/log/auth.log | tail -10

# Invalid users attempting login
grep "Invalid user" /var/log/auth.log | tail -10
```

### Connection Analysis  
```bash
# Current SSH connections
ss -tnp | grep :2222

# Active SSH sessions
who -u

# Login history
last -n 20

# Failed login attempts by IP
grep "Failed password" /var/log/auth.log | \
  awk '{print $(NF-3)}' | sort | uniq -c | sort -nr
```

### Brute Force Detection
```bash
# Show IPs with multiple failed attempts
grep "Failed password" /var/log/auth.log | \
  awk '{print $(NF-3)}' | sort | uniq -c | \
  awk '$1 > 5 {print $2 " (" $1 " attempts)"}'

# Recent failed attempts (last hour)
grep "$(date '+%b %d %H:')" /var/log/auth.log | \
  grep "Failed password"

# Successful logins after failed attempts (potential compromise)
grep -A5 -B5 "Accepted" /var/log/auth.log | \
  grep -E "Failed password|Accepted"
```

## Network-Level Monitoring

### Port Scanning Detection
```bash
# Check for connection attempts to common ports
ss -tlnp | grep -E ":(22|2222|2223|80|443|8080)"

# Monitor connection attempts (requires netstat)
netstat -tln | grep LISTEN | grep -E ":(22|2222|2223)"

# Check iptables logs (if logging enabled)
grep "DROP" /var/log/kern.log | tail -10
```

### Active Connection Monitoring
```bash
# Show all network connections
ss -tuln

# Monitor new connections
watch -n 2 'ss -tn | grep :2222'

# Check for unusual processes using network
lsof -i :2222
lsof -i :22
```

## Alerting and Notifications

### Simple Alert Scripts
Create monitoring scripts for common scenarios:

```bash
# Alert on honeypot activity
#!/bin/bash
# /usr/local/bin/honeypot-alert.sh
LAST_CHECK="/tmp/honeypot-last-check"
LOG_FILE="/var/log/honeypot.log"

if [ ! -f "$LAST_CHECK" ]; then
    touch "$LAST_CHECK"
fi

NEW_ATTACKS=$(find "$LOG_FILE" -newer "$LAST_CHECK" | wc -l)
if [ "$NEW_ATTACKS" -gt 0 ]; then
    echo "ALERT: $NEW_ATTACKS new honeypot attacks detected"
    tail -n "$NEW_ATTACKS" "$LOG_FILE"
fi

touch "$LAST_CHECK"
```

### Fail2ban Integration
Monitor fail2ban status for automatic IP blocking:

```bash
# Check fail2ban status
sudo fail2ban-client status

# Check SSH jail specifically  
sudo fail2ban-client status sshd

# Show banned IPs
sudo fail2ban-client get sshd banip

# Unban IP if needed
sudo fail2ban-client set sshd unbanip 192.168.1.100
```

## Log Analysis Tools

### Basic Analysis Commands
```bash
# Most common attacking IPs across both services
(grep "honeypot connection" /var/log/honeypot.log | awk '{print $NF}'; \
 grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}') | \
 sort | uniq -c | sort -nr | head -10

# Timeline of attacks (both honeypot and real SSH)
(grep "honeypot connection" /var/log/honeypot.log | sed 's/honeypot/HONEYPOT/'; \
 grep "Failed password" /var/log/auth.log | sed 's/Failed password/REAL_SSH/') | \
 sort -k1,3
```

### Advanced Analysis
```bash
# Attack patterns by time of day
grep -E "(honeypot connection|Failed password)" \
  /var/log/honeypot.log /var/log/auth.log | \
  awk '{print $4}' | cut -d: -f1 | sort | uniq -c | \
  sort -k2 -n

# Correlation between honeypot and real SSH attacks
comm -12 \
  <(grep "honeypot connection" /var/log/honeypot.log | awk '{print $NF}' | sort -u) \
  <(grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort -u)
```

## Incident Response

### Immediate Response
```bash
# Block suspicious IP immediately
sudo iptables -A INPUT -s SUSPICIOUS_IP -j DROP

# Check current connections from IP
ss -tn | grep SUSPICIOUS_IP

# Kill any active sessions from IP
sudo pkill -f "sshd.*SUSPICIOUS_IP"
```

### Investigation Steps
1. **Identify Attack Source**: Analyze logs to determine origin IP and attack pattern
2. **Check Compromise**: Look for successful logins after failed attempts  
3. **Assess Impact**: Check for file modifications, new users, or privilege escalation
4. **Implement Blocks**: Add IP to fail2ban or firewall rules
5. **Monitor**: Continue monitoring for related activity

### Forensic Analysis
```bash
# Check for privilege escalation attempts
grep -i "sudo\|su\|root" /var/log/auth.log | tail -20

# Look for file access patterns
find /var/log -name "*.log" -mtime -1 -exec grep "SUSPICIOUS_IP" {} \;

# Check for new user accounts or modifications
grep -E "(useradd|usermod|passwd)" /var/log/auth.log | tail -10
```

## Automated Monitoring Setup

### Cron Jobs for Regular Monitoring
```bash
# Add to crontab (crontab -e)
# Check for new attacks every 5 minutes
*/5 * * * * /usr/local/bin/honeypot-alert.sh

# Daily security report
0 9 * * * /usr/local/bin/daily-security-report.sh

# Weekly log cleanup
0 2 * * 0 /usr/local/bin/cleanup-old-logs.sh
```

### System Integration
- Configure rsyslog to separate security logs
- Set up log rotation for security logs
- Integrate with monitoring systems (Nagios, Zabbix, etc.)
- Configure email alerts for critical events

## Best Practices

### Monitoring Frequency
- **Real-time**: Honeypot attacks, SSH authentication failures
- **Hourly**: Connection pattern analysis, unusual activity
- **Daily**: Attack summary, trend analysis, IP reputation checks
- **Weekly**: Comprehensive security review, log archiving

### Log Retention
- **Honeypot logs**: 30 days for analysis, 90 days archived
- **SSH auth logs**: 90 days active, 1 year archived  
- **Security logs**: 180 days active, 2 years archived
- **Incident logs**: Permanent retention for forensic analysis

### Performance Considerations
- Monitor log file sizes and implement rotation
- Use efficient grep/awk patterns for large log files
- Consider log aggregation tools for high-volume environments
- Archive old logs to prevent disk space issues