Log Ingestor¶

The log ingestor (mailer/log_ingestor/, added 2026-08-22) tails Postfix's mail log and turns it into per-message rows in the mail_logs table, firing the matching email.delivered / email.bounced / email.deferred / email.rejected webhooks as it goes. It is the sole producer for mail_logs -- and, through the webhooks it fires at mailyte-api, for that system's delivery_events.

Why It Exists¶

Before this container, nothing in the stack ever wrote a per-message row to mail_logs. The table, the platform API endpoints that read it (message_trace, analytics), and the tenant Email Logs UI all existed around a producer that had never been built -- confirmed on production 2026-08-22, where mail_logs sat at 0 rows while mail flowed normally.

The components that looked like the producer are not:

• mailer/log_analyzer/ writes aggregate mail_analysis_reports, never per-message rows -- and is in no compose file, so no container even exists
• webhook_sender.py is defined in master.cf as the webhook-filter transport, but no content_filter references it -- dead config
• tracking_injector.py has a webhook hook that returns immediately unless TRACKING_WEBHOOK_URL is set, which it is not

Postfix's own log is the right source: it is the authoritative record of every delivery attempt regardless of how the message was submitted (SMTP, submission, webmail, API), and mail_logs' columns -- relay, delays, dsn, status -- map onto a Postfix delivery line one-for-one.

How It Works¶

flowchart LR
    PF["Postfix"] -->|writes| Log["./logs/mailer/postfix/mail.log"]
    LI["log_ingestor\n(tails read-only)"] --> Log
    LI -->|"INSERT IGNORE"| ML[("mail_logs")]
    LI -->|"email.delivered / bounced /\ndeferred / rejected"| WH["shared.webhook_dispatcher\n→ WEBHOOK_URL (mailyte-api)"]
    LI -->|"offset + inode"| State[("log_ingestor_state volume")]
    LI -.->|"auto-suspend (K2, off by default)"| Doveadm["dovecot doveadm HTTP :24180"]

• Idempotent by construction: each row's primary key is derived deterministically from (queue id, recipient, status, timestamp) and inserted with INSERT IGNORE. Re-reading a log region after a crash or replay cannot create duplicates.
• Rotation-safe: rotation is detected by inode, not size, so a rotated-then-smaller file is not mistaken for truncation.
• Filter hops are excluded: Postfix logs a status=sent line when it hands a message to the tracking filter and another when the re-injected copy is delivered. Relays listed in INGESTOR_INTERNAL_RELAYS (default tracking-filter,webhook-filter) are skipped so each message is counted once.
• Auto-suspend (SMTP API keys K2): with AUTO_SUSPEND_ENABLED=true, an SMTP API key whose recent traffic exceeds a failure ratio is suspended and Dovecot's auth cache is flushed through the doveadm HTTP API so the suspension bites immediately. Off by default until the thresholds have been observed against real traffic.

Invocation¶

Not an HTTP service -- it is a long-running tailer with no listening port, started by compose:

log_ingestor:
  build:
    context: .
    dockerfile: ./mailer/log_ingestor/Dockerfile
  container_name: log_ingestor
  volumes:
    - ./logs/mailer/postfix:/var/log/postfix:ro    # postfix owns the file; we only follow it
    - log_ingestor_state:/var/lib/log_ingestor

The state volume keeps the read offset so a recreate doesn't re-read the whole log (a replay would be harmless -- just wasteful -- thanks to the deterministic keys).

Configuration¶

Data Written¶

Gotchas¶