Log Analyzer -- Real-Time Log Intelligence¶

The log analyzer is a Python service that continuously processes mail server logs, detects patterns, computes statistics, and feeds alerts into the monitoring system. While Fail2ban handles reactive blocking, the log analyzer provides the big-picture view -- delivery rates, bounce trends, spam volumes, geographic data, and anomaly detection.

What It Does¶

• Parses Postfix, Dovecot, and Rspamd logs in real time
• Computes delivery statistics (sent, bounced, deferred, rejected)
• Tracks top sender/receiver domains
• Detects anomalies (sudden spike in bounces, unusual sending patterns)
• Provides geographic analysis of connecting IPs (via GeoIP)
• Stores aggregated stats in MySQL for the analytics worker
• Sends webhook alerts for detected anomalies

How It Works¶

flowchart LR
    subgraph Logs
        PF["/var/log/mail.log"]
        DC["/var/log/dovecot.log"]
        RS["/var/log/rspamd/rspamd.log"]
    end

    subgraph Analyzer["Log Analyzer"]
        Parser["Pattern Matcher"]
        Stats["Statistics\nAggregator"]
        Anomaly["Anomaly\nDetector"]
        Geo["GeoIP\nResolver"]
    end

    PF --> Parser
    DC --> Parser
    RS --> Parser

    Parser --> Stats
    Parser --> Anomaly
    Parser --> Geo

    Stats --> MySQL[(MySQL)]
    Anomaly --> Webhook["Webhook Alert"]
    Geo --> MySQL

Log Patterns¶

The analyzer uses compiled regex patterns to parse log lines. Here are the patterns it recognizes:

Postfix Patterns¶

Dovecot Patterns¶

Rspamd Patterns¶

Statistics Computed¶

The analyzer computes rolling statistics over configurable time windows (default: last 24 hours):

Delivery Stats¶

• Total emails sent, bounced, deferred, rejected
• Delivery success rate (%)
• Bounce rate by category (hard/soft)
• Deferred queue growth rate

Domain Analytics¶

• Top 10 sending domains
• Top 10 receiving domains
• Per-domain delivery rates

Spam Stats¶

• Total spam detected vs. ham
• Spam ratio over time
• Top spam sources (IP/domain)

Security Stats¶

• Failed login attempts by IP
• Successful logins by IP and user
• Geographic distribution of connections

Hourly Volume¶

• Emails per hour breakdown
• Peak hour identification
• Trend analysis (increasing/decreasing)

Anomaly Detection¶

The analyzer watches for patterns that indicate problems:

When an anomaly is detected, the analyzer fires a webhook event and logs the alert. The monitoring dashboard displays these alerts in real time.

GeoIP Integration¶

If the GeoLite2 database is available, the analyzer resolves connecting IPs to countries:

self.geoip_db = "/usr/share/GeoIP/GeoLite2-Country.mmdb"

This data feeds into:

• Geographic heatmaps in the dashboard
• Country-based anomaly detection (sudden traffic from a new country)
• The analytics worker for per-organization geo reports

Database Tables¶

The analyzer writes aggregated stats to:

Configuration¶

Adding Custom Rules¶

To add a new detection pattern, add a compiled regex to the patterns dictionary in mailer/log_analyzer/app.py:

self.patterns["my_custom_pattern"] = re.compile(
    r"my-service\[\d+\]: suspicious activity from \[([^\]]+)\]"
)

Then add handling logic in the appropriate analysis method to extract data from the matched groups and record it.

Gotchas¶