Rate Limiter Worker¶

The rate limiter worker provides a centralized, Redis-backed rate limiting service for the entire mail system. It enforces sending limits at the organization, domain, and mailbox level, and integrates with Postfix's policy service for real-time enforcement during SMTP transactions.

There are actually two rate limiting components:

1. Postfix policy script (mailer/postfix/scripts/rate_limit_policy.py) -- runs inside the Postfix container, does real-time checks during SMTP
2. Rate limiter worker (worker/rate_limiter/) -- the standalone service with an API for managing limits, viewing usage, and alerting

They share the same Redis counters, so limits are enforced consistently regardless of which component checks them.

What It Does¶

• Sliding window rate limiting using Redis atomic counters
• Three-tier hierarchy: Organization > Domain > Mailbox
• Configurable limits stored in MySQL settings JSON column
• Real-time usage tracking and quota consumption reporting
• Threshold alerts via webhooks when approaching limits
• Prometheus metrics for monitoring
• API for querying and updating rate limits

How It Works¶

flowchart TB
    subgraph Postfix
        SMTP["SMTP Transaction"]
        Policy["rate_limit_policy.py"]
    end

    subgraph Worker["Rate Limiter Worker :8082"]
        API["Flask API"]
        ConfigSvc["Config Service"]
        CacheSvc["Cache Service"]
        DBSvc["Database Service"]
        UsageSvc["Usage Service"]
        AlertSvc["Alert Service"]
        WebhookSvc["Webhook Service"]
    end

    SMTP -->|"policy check"| Policy
    Policy -->|"INCR + check"| Redis[(Redis)]
    Policy -->|"get limits"| MySQL[(MySQL)]

    API --> ConfigSvc
    API --> UsageSvc
    CacheSvc --> Redis
    DBSvc --> MySQL
    AlertSvc --> WebhookSvc

Rate Limit Hierarchy¶

Organization (default: 10,000/hour)
  └── Domain (default: 5,000/hour)
       └── Mailbox (default: 1,000/hour)

All three levels are checked. If any level is exceeded, the email is rejected with:

REJECT 4.7.1 Rate limit exceeded for {entity}. Try again later.

The 4.7.1 code is a temporary failure, so the sending client will retry.

Redis Key Structure¶

rate:{entity_type}:{identifier}:{YYYY-MM-DD-HH}

Examples:

rate:mailbox:user@example.com:2025-01-15-14
rate:domain:example.com:2025-01-15-14
rate:organization:org-uuid:2025-01-15-14

Keys expire after 2 hours (7200 seconds) so old counters are automatically cleaned up.

Custom Limits¶

Limits are stored as JSON in the settings column of the respective table:

-- Set a custom limit for a mailbox
UPDATE email_accounts
SET settings = JSON_SET(COALESCE(settings, '{}'), '$.rate_limits.hourly', 500)
WHERE email = 'user@example.com';

-- Set a custom limit for a domain
UPDATE domains
SET settings = JSON_SET(COALESCE(settings, '{}'), '$.rate_limits.hourly', 2000)
WHERE domain = 'example.com';

If no custom limit is set, the defaults apply.

API Endpoints¶

GET  /api/rate-limits/{entity_type}/{identifier}  -- Current usage and limits
PUT  /api/rate-limits/{entity_type}/{identifier}  -- Update limits
GET  /api/rate-limits/status                       -- Overview of all active limiters
GET  /health                                        -- Health check
GET  /metrics                                       -- Prometheus metrics

Configuration¶

Docker Configuration¶

rate_limiter:
  build: ./worker/rate_limiter
  container_name: rate_limiter
  ports:
    - "8082:8082"
  depends_on:
    - mysql
    - redis

Gotchas¶