Skip to main content

Dead-letter Queue Monitor

Overview​

When a message has exhausted its 3 retries or was classified as a permanent error from the start, it is dead-lettered into <queue>.dlq. Left unattended, those messages simply pile up unnoticed.

The DLQ Monitor is a dedicated consumer running on the main profile. It subscribes to every monitored DLQ, logs the full detail of each dead message (payload, x-death, x-retry-count), and batches those events into chat webhook alerts with a cooldown so the channel is never spammed.

Business Flow​

  1. When runMain starts, it constructs a DLQMonitor and runs it as a goroutine separate from the main registrar.
  2. The monitor opens one consumer per DLQ, following the MonitoredDLQQueues list of 37 queues — all of AllQueues except message_received_trigger, which is intentionally excluded.
  3. Each time a message lands in a DLQ, the monitor:
    • Unpacks and logs the source queue name, the payload, the error, and the x-death and x-retry-count headers
    • Appends an entry to pendingAlerts recording the queue, payload, and error
  4. Once DLQ_ALERT_COOLDOWN_MS (5 minutes by default) has elapsed since the previous alert, all pending entries are combined into a single message, sent to DLQ_ALERT_WEBHOOK_URL, and the buffer is cleared.
  5. The monitor always acks, whether or not the payload parsed successfully, so that no endless redelivery loop can form inside the DLQ.

Key Files & Functions​

  • internal/mq/dlqmonitor.go
    • Client.NewDLQMonitor() — builds the monitor from configuration (DLQ_ALERT_WEBHOOK_URL, DLQ_ALERT_COOLDOWN_MS)
    • DLQMonitor.Run(ctx) — starts consumers for every DLQ concurrently
    • DLQMonitor.consume(ctx, dlqQueue, suffix) — the consume loop for a single DLQ
    • dlqAlert — the struct representing one alert entry
  • internal/mq/topology.go — MonitoredDLQQueues(), the set of monitored queues
  • cmd/worker/main.go — the part of runMain() that constructs and starts the monitor
  • Subscribed queues are every <queue>.dlq, where the suffix comes from RABBITMQ_DLQ_SUFFIX (default .dlq), all bound to the line_exchange_dlq exchange

Connections to Other Services​

  • RabbitMQ — consumes from all DLQs; the topology must be asserted first
  • Chat webhook (DLQ_ALERT_WEBHOOK_URL) — an HTTP POST with a {"text": ...} body, compatible with both Google Chat and Slack, using a 5-second client timeout. If the URL is not configured, the monitor logs only and sends no alerts.
  • No database access — this feature is purely observability
  • Directly related to RabbitMQ Queue Topology & Retry / Dead-letter Mechanics