Monitoring Service¶

The monitoring service is the watchdog for the entire Mailyte stack. It continuously checks the health of every service, tracks system resources, triggers auto-healing when things go wrong, and sends alerts via webhooks. It also provides a web dashboard and Prometheus metrics endpoint.

What It Does¶

• System resource monitoring: CPU, memory, disk usage
• Service health checks: SMTP, IMAP, HTTP workers, Redis, MySQL
• Automated recovery: restarts failed services via Docker
• Webhook notifications: alerts on service failures and recoveries
• Prometheus metrics: exports gauges, counters, and histograms
• Admin API: manual service control and status queries
• Scheduling: periodic health checks on configurable intervals

How It Works¶

flowchart TB
    subgraph Monitor["Monitoring Service :8085"]
        Scheduler["Schedule\nLoop"]
        SysMon["System Monitor\n(CPU/RAM/Disk)"]
        SvcMon["Service Monitor\n(health checks)"]
        WebhookN["Webhook\nNotifier"]
        API["Flask API"]
        Prom["Prometheus\nMetrics"]
    end

    Scheduler --> SysMon
    Scheduler --> SvcMon

    SvcMon -->|"SMTP connect"| Postfix
    SvcMon -->|"IMAP connect"| Dovecot
    SvcMon -->|"HTTP /health"| Workers["Workers"]
    SvcMon -->|"PING"| Redis[(Redis)]
    SvcMon -->|"SELECT 1"| MySQL[(MySQL)]

    SvcMon -->|"failure detected"| WebhookN
    SvcMon -->|"auto-heal"| Docker["Docker API\n(restart)"]

    API --> SysMon
    API --> SvcMon
    Prom --> SysMon
    Prom --> SvcMon

Health Check Methods¶

Each service type has its own check:

Prometheus Metrics¶

The service exports these metrics:

Access metrics at GET /metrics in Prometheus exposition format.

Auto-Healing¶

When a service fails its health check multiple times in a row, the monitor can automatically restart it:

1. First failure: log a warning, send webhook alert
2. Second consecutive failure: attempt Docker container restart
3. If restart succeeds: send recovery webhook
4. If restart fails: escalate to critical alert

Auto-healing requires Docker socket access (/var/run/docker.sock).

API Endpoints¶

GET  /health                          -- Overall system health summary
GET  /api/status                      -- Detailed status of all services
GET  /api/status/{service_name}       -- Status of a specific service
POST /api/services/{name}/restart     -- Manually restart a service
GET  /api/system                       -- System resource metrics
GET  /api/alerts                       -- Recent alerts
GET  /metrics                          -- Prometheus metrics

System Health Response¶

{
  "status": "healthy",
  "services": {
    "postfix": {"status": "up", "latency_ms": 12},
    "dovecot": {"status": "up", "latency_ms": 8},
    "rspamd": {"status": "up", "latency_ms": 25},
    "mysql": {"status": "up", "latency_ms": 3},
    "redis": {"status": "up", "latency_ms": 1}
  },
  "system": {
    "cpu_percent": 35.2,
    "memory_percent": 62.8,
    "disk_percent": 45.1
  }
}

Service Architecture¶

Configuration¶

Database Tables¶

Docker Configuration¶

monitoring:
  build: ./worker/monitoring
  container_name: monitoring
  ports:
    - "8085:8085"
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock  # For auto-healing
  depends_on:
    - mysql
    - redis

Gotchas¶