Queues · Lesson 4
Lesson 2 left a loose thread: what stops a message that can never succeed from looping forever? The answer is the dead-letter queue — and the way conveyor tunes it exposes a deep, non-obvious coupling between failure handling and backpressure. This is themaxReceiveCount: 20in yourmain.go, finally explained.
Every SQS message carries an ApproximateReceiveCount — how many times it has been delivered to a consumer. SQS bumps it on every receive. A queue can carry a redrive policy: a dead-letter queue plus a maxReceiveCount. The rule is one line:
| Queue | maxReceiveCount | Dead-letters to | DLQ retention |
|---|---|---|---|
per-run task (…-run-{id}) | 20 | shared …-tasks-dlq | 14 days |
control (…-control) | 5 | …-control-dlq | 14 days |
Two design notes you can read straight off the infra. First, one shared
task DLQ serves every per-run queue (sqs.tf: "a single DLQ
keeps the resource graph bounded") — with a fresh queue created per run, you do
not want a DLQ per run too. Second, the DLQs keep messages 14
days versus 4 on the live queues: a dead-lettered message is a thing a
human needs time to find and triage.
The control queue uses the conventional maxReceiveCount: 5 — five
delivery attempts, then quarantine. So does the default in
perjob.go. But main.go overrides the per-run queues to
20, with this comment:
// COR-314: per-job concurrency throttling can cycle a message
// through a handful of SQS receives even with the dispatcher
// cooldown in place; 20 gives ~100s of sustained-throttle
// headroom before redrive instead of the default 5.
MaxReceiveCount: 20,maxReceiveCount: 5, a busy job's good tasks would get dead-lettered
for the crime of being throttled five times. Raising it to 20 buys headroom so
only genuinely stuck messages reach the DLQ. The DLQ threshold and the
flow-control design are coupled — you can't size one without the
other.
That coupling is the real lesson. maxReceiveCount looks like "how
many retries before giving up," but it's really "how many redeliveries
from any cause." If you re-receive messages for non-failure reasons, you
must budget for them — which is exactly why Lesson 5's bounce also backs off
exponentially, to spend that budget as slowly as possible.
Recall Lesson 2's rule: return nil for poison (settle and drop),
error only for transient failures. The DLQ is the backstop for
the poison you didn't predict — a bug that makes some messages always
error will loop, climb to maxReceiveCount, and land in the DLQ where
you'll see it, instead of silently retrying forever. A spiking DLQ is one
of the highest-signal alerts an SQS system has.
giveUp helper), with a
TODO(v1.1) to wire sqs:SendMessage on the DLQ. So today
the control DLQ only catches messages that hit maxReceiveCount: 5 via
redelivery, not deliberate give-ups. A useful reminder: a DLQ catches what
redelivery exhausts; routing chosen failures there is a separate
wiring you have to do on purpose.
DLQ + maxReceiveCount is a first-class, declarative feature. Redrive back to source is a button.
Dead-lettering via a per-queue DLX (dead-letter exchange); the "count" is
your own x-delivery-count / header bookkeeping.
No native DLQ. You build one: catch, produce to a *.DLT topic
yourself (Spring Kafka, Connect, etc. ship helpers). It's a pattern, not a
primitive.
Primary source: AWS
· Amazon SQS dead-letter queues (and the linked "redrive" page — moving
messages back to source after you've fixed the bug). Then read
infra/lambda/sqs.tf and the MaxReceiveCount: 20 block in
cmd/conveyor/main.go with this lesson open.
Want to walk through what you'd actually do when the tasks-dlq alarm fires — inspect, fix, redrive — or how to set a DLQ alarm? Ask me.