Queues · Lesson 6 · a real incident

How conveyor "froze": receive amplification, seen live

You shipped the fix (COR-316) but wanted the intuition: how could conveyor freeze at all? This is the capstone of Lessons 3–5 — the concurrency limit, the bounce, and long polling all collide here. And there's a simulator below so you can watch it happen, then flip one switch and watch it stop.
First, the reassurance Nothing ever deadlocked. The "freeze" is throughput collapse on a concurrency-limited job: work stops flowing even though every worker is busy. The codebase's name for the cause is receive amplification — amplification = receives ÷ completions. When that ratio blows up, the job crawls and valid tasks leak to the DLQ.

Watch it: run the simulator

One hot run-queue, 10 tasks, a pool of workers, and a per-job limit of 1 permit (only one task may process at a time). Press Run with the gate OFF and watch the Amplification and Lost to DLQ numbers. Then — even mid-run — click Pre-poll gate: OFF to flip it ON and watch the behaviour change.

With the gate off, the queue empties into the "bounced" lane, workers keep receiving-and-bouncing, receives sprint ahead of completions, and tasks creep toward the DLQ. With the gate on, idle workers go 🔒 gated while the permit is held — they don't even receive — so receives track completions and nothing is lost. Same code, one decision moved earlier.

Why it happens — three ingredients you already know

1
A per-job concurrency limit (ADR 005). The job allows 1 task in flight; the permit is held for the whole scrape — possibly minutes (L5).
2
A pool of 100 independent workers each doing pick → receive → process, none coordinating (L3).
3
The bounce throttled tasks are re-hidden and come back — and every bounce is a receive (L4).

The flaw tying them together: nothing stopped a worker from polling a queue whose job was already maxed out. So while one worker holds the single permit and scrapes, the other 99 keep receiving the queued tasks and bouncing every one. Thousands of receives, near-zero completions — that ratio is the amplification. In prod it showed up as the CloudWatch NumberOfMessagesReceived / Deleted ratio exploding.

That one flaw radiates into three symptoms:

Symptom 1 · throttle → DLQ loss Every bounce bumps ApproximateReceiveCount. Enough bounces during one long scrape and a perfectly valid, just-waiting task crosses maxReceiveCount and gets dead-lettered (L4). That's why COR-314 raised it to 20 and COR-315 made bounces back off exponentially — mitigations that slow the bleed, not the cure.
Symptom 2 · the "frozen" feel After a bounce, the queue goes on a 5s cooldown, so pickRandom finds nothing pickable and the whole pool sleeps. Bounced tasks reappear on ever-growing visibility (5s→10s→20s…). The job's work trickles out in lurches with long dead gaps. From the outside: frozen.
Symptom 3 · cost storm The same wasted receives, before long polling, were the empty-receive bill from L3 — hundreds of dollars/day of pure waste.

The twist: fixing one thing exposed it

Short polling masked this. Receives returned instantly, the pool churned fast, and the amplification hid inside the empty-receive cost storm. The moment that storm was fixed by switching to long polling (ff44a57), the dynamics shifted: a worker that picks a maxed-out queue now blocks up to 20s, tying up pool capacity, and the stall became plain. A classic sequence — you fix the visible problem and reveal the one hiding beneath it.

The fix you shipped: the pre-poll gate (COR-316)

One idea: don't fetch work you can't run. pickRandom now checks the job's local in-flight count before polling, and skips the queue if it's already at the limit: 

// dispatcher.go — pickRandom, COR-316
if entry.maxInflight > 0 &&
   int(entry.localInflight.Load()) >= entry.maxInflight {
    continue   // job is maxed — don't even Receive; we'd only bounce it
}

While the permit is held, the queue is gated off: zero receives, zero bounces, zero amplification. When the task finishes and the permit frees, the gate opens and one worker picks up the next task. Your test TestCOR316_GateReducesReceiveAmplification nails it: gate-on ⇒ strictly fewer receives, ≥ completions, ≤ DLQ.

Two honest caveats The gate reads the dispatcher's local count; the DynamoDB Limiter stays the cross-container source of truth. With 10 worker containers and a limit of 1, each container's gate allows 1 local attempt → up to 10 globally, and the limiter throttles the rest. So the gate doesn't eliminate amplification across containers — it collapses it from "every worker in every container" to "one worker per container." And it's an optimisation layer: the limiter still enforces correctness; the gate just stops the pointless receive.
The one-line takeaway The freeze is what happens when backpressure is "receive, then reject" instead of "don't receive." Bouncing (L5) handles the occasional overflow fine — but if the pool keeps fetching work it must immediately bounce, you get amplification. The gate moves the decision earlier. That's almost always where backpressure belongs: at intake, not after.
Aside · the other "latest commit" (#91) is not freeze logic

0b5675a is a merge-order build break, not a behaviour change. PR #83 (the gate) widened addLocked to 3 args; PR #88 (long-poll), branched before #83 merged, added a 2-arg call in a test. The textual merge compiled neither side's intent and broke main's build; #91 passes 0 (gate disabled) to that test call to restore it. And 2358435 just reverts DefaultMaxInflight back to 500 now the freeze loadtest (which forced it to 1) is done.

Read this next

Your own primary sources: docs/adrs/005-per-job-concurrency-limit.md, the commit trail (ff44a57 long-poll → 2caad22 COR-314 → 731dede COR-315 → b19f72e COR-316), and the local reproduction in cmd/conveyor-loadtest/main.go + internal/harness/harness.go (which prints the very amplification number this lesson visualises).

Want me to trace one specific run through the loadtest harness, or to add a second slider to the simulator (more permits, more workers) so you can feel how the amplification scales? Ask me.

← Lesson 5 · Control plane Next → The dynamic queue-per-run architecture