homelab/docs/ssh-intrusion-monitoring.md

SSH Intrusion Monitoring Guide

Overview

This guide covers comprehensive SSH intrusion detection and monitoring for the homelab environment, including both honeypot analysis and real SSH service monitoring.

Log Files and Locations

Primary Log Files

• Honeypot Logs: /var/log/honeypot.log - Attack attempts on port 22
• SSH Authentication: /var/log/auth.log - Real SSH attempts on port 2222
• System Security: /var/log/secure or /var/log/security.log - General security events
• Service Logs: journalctl -u sshd.service - SSH daemon logs

Service Status Monitoring

# Check honeypot service
systemctl status ssh-honeypot.service

# Check real SSH service  
systemctl status sshd.service

# Monitor both services
systemctl status ssh-honeypot.service sshd.service

Honeypot Monitoring

Real-time Monitoring

# Live honeypot attack monitoring
tail -f /var/log/honeypot.log

# Monitor with timestamps
tail -f /var/log/honeypot.log | while read line; do echo "$(date '+%H:%M:%S') $line"; done

Attack Analysis

# Count total attack attempts
grep -c "honeypot connection" /var/log/honeypot.log

# Show unique attacking IP addresses with attempt counts
grep "honeypot connection" /var/log/honeypot.log | \
  awk '{print $NF}' | sort | uniq -c | sort -nr

# Attacks by hour of day
grep "honeypot connection" /var/log/honeypot.log | \
  awk '{print $4}' | cut -d: -f1 | sort | uniq -c | sort -k2 -n

# Recent attacks (last 24 hours)
grep "$(date '+%a %b %d')" /var/log/honeypot.log

# Attacks from specific IP
grep "192.168.1.100" /var/log/honeypot.log

Geographic Analysis

# Get country information for attacking IPs (requires geoip)
grep "honeypot connection" /var/log/honeypot.log | \
  awk '{print $NF}' | sort -u | \
  while read ip; do
    echo -n "$ip: "
    geoiplookup "$ip" 2>/dev/null | head -1 | cut -d: -f2
  done

Real SSH Monitoring

Authentication Monitoring

# Monitor real SSH authentication attempts
tail -f /var/log/auth.log | grep sshd

# Failed password attempts
grep "Failed password" /var/log/auth.log | tail -10

# Successful logins
grep "Accepted password\|Accepted publickey" /var/log/auth.log | tail -10

# Invalid users attempting login
grep "Invalid user" /var/log/auth.log | tail -10

Connection Analysis

# Current SSH connections
ss -tnp | grep :2222

# Active SSH sessions
who -u

# Login history
last -n 20

# Failed login attempts by IP
grep "Failed password" /var/log/auth.log | \
  awk '{print $(NF-3)}' | sort | uniq -c | sort -nr

Brute Force Detection

# Show IPs with multiple failed attempts
grep "Failed password" /var/log/auth.log | \
  awk '{print $(NF-3)}' | sort | uniq -c | \
  awk '$1 > 5 {print $2 " (" $1 " attempts)"}'

# Recent failed attempts (last hour)
grep "$(date '+%b %d %H:')" /var/log/auth.log | \
  grep "Failed password"

# Successful logins after failed attempts (potential compromise)
grep -A5 -B5 "Accepted" /var/log/auth.log | \
  grep -E "Failed password|Accepted"

Network-Level Monitoring

Port Scanning Detection

# Check for connection attempts to common ports
ss -tlnp | grep -E ":(22|2222|2223|80|443|8080)"

# Monitor connection attempts (requires netstat)
netstat -tln | grep LISTEN | grep -E ":(22|2222|2223)"

# Check iptables logs (if logging enabled)
grep "DROP" /var/log/kern.log | tail -10

Active Connection Monitoring

# Show all network connections
ss -tuln

# Monitor new connections
watch -n 2 'ss -tn | grep :2222'

# Check for unusual processes using network
lsof -i :2222
lsof -i :22

Alerting and Notifications

Simple Alert Scripts

Create monitoring scripts for common scenarios:

# Alert on honeypot activity
#!/bin/bash
# /usr/local/bin/honeypot-alert.sh
LAST_CHECK="/tmp/honeypot-last-check"
LOG_FILE="/var/log/honeypot.log"

if [ ! -f "$LAST_CHECK" ]; then
    touch "$LAST_CHECK"
fi

NEW_ATTACKS=$(find "$LOG_FILE" -newer "$LAST_CHECK" | wc -l)
if [ "$NEW_ATTACKS" -gt 0 ]; then
    echo "ALERT: $NEW_ATTACKS new honeypot attacks detected"
    tail -n "$NEW_ATTACKS" "$LOG_FILE"
fi

touch "$LAST_CHECK"

Fail2ban Integration

Monitor fail2ban status for automatic IP blocking:

# Check fail2ban status
sudo fail2ban-client status

# Check SSH jail specifically  
sudo fail2ban-client status sshd

# Show banned IPs
sudo fail2ban-client get sshd banip

# Unban IP if needed
sudo fail2ban-client set sshd unbanip 192.168.1.100

Log Analysis Tools

Basic Analysis Commands

# Most common attacking IPs across both services
(grep "honeypot connection" /var/log/honeypot.log | awk '{print $NF}'; \
 grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}') | \
 sort | uniq -c | sort -nr | head -10

# Timeline of attacks (both honeypot and real SSH)
(grep "honeypot connection" /var/log/honeypot.log | sed 's/honeypot/HONEYPOT/'; \
 grep "Failed password" /var/log/auth.log | sed 's/Failed password/REAL_SSH/') | \
 sort -k1,3

Advanced Analysis

# Attack patterns by time of day
grep -E "(honeypot connection|Failed password)" \
  /var/log/honeypot.log /var/log/auth.log | \
  awk '{print $4}' | cut -d: -f1 | sort | uniq -c | \
  sort -k2 -n

# Correlation between honeypot and real SSH attacks
comm -12 \
  <(grep "honeypot connection" /var/log/honeypot.log | awk '{print $NF}' | sort -u) \
  <(grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort -u)

Incident Response

Immediate Response

# Block suspicious IP immediately
sudo iptables -A INPUT -s SUSPICIOUS_IP -j DROP

# Check current connections from IP
ss -tn | grep SUSPICIOUS_IP

# Kill any active sessions from IP
sudo pkill -f "sshd.*SUSPICIOUS_IP"

Investigation Steps

1. Identify Attack Source: Analyze logs to determine origin IP and attack pattern
2. Check Compromise: Look for successful logins after failed attempts
3. Assess Impact: Check for file modifications, new users, or privilege escalation
4. Implement Blocks: Add IP to fail2ban or firewall rules
5. Monitor: Continue monitoring for related activity

Forensic Analysis

# Check for privilege escalation attempts
grep -i "sudo\|su\|root" /var/log/auth.log | tail -20

# Look for file access patterns
find /var/log -name "*.log" -mtime -1 -exec grep "SUSPICIOUS_IP" {} \;

# Check for new user accounts or modifications
grep -E "(useradd|usermod|passwd)" /var/log/auth.log | tail -10

Automated Monitoring Setup

Cron Jobs for Regular Monitoring

# Add to crontab (crontab -e)
# Check for new attacks every 5 minutes
*/5 * * * * /usr/local/bin/honeypot-alert.sh

# Daily security report
0 9 * * * /usr/local/bin/daily-security-report.sh

# Weekly log cleanup
0 2 * * 0 /usr/local/bin/cleanup-old-logs.sh

System Integration

• Configure rsyslog to separate security logs
• Set up log rotation for security logs
• Integrate with monitoring systems (Nagios, Zabbix, etc.)
• Configure email alerts for critical events

Best Practices

Monitoring Frequency

• Real-time: Honeypot attacks, SSH authentication failures
• Hourly: Connection pattern analysis, unusual activity
• Daily: Attack summary, trend analysis, IP reputation checks
• Weekly: Comprehensive security review, log archiving

Log Retention

• Honeypot logs: 30 days for analysis, 90 days archived
• SSH auth logs: 90 days active, 1 year archived
• Security logs: 180 days active, 2 years archived
• Incident logs: Permanent retention for forensic analysis

Performance Considerations

• Monitor log file sizes and implement rotation
• Use efficient grep/awk patterns for large log files
• Consider log aggregation tools for high-volume environments
• Archive old logs to prevent disk space issues