Queues · Lesson 5 · the payload

Visibility as a control plane: building flow control out of a clock

This is the lesson you actually came for. SQS gives you no scheduler, no priorities, no "deliver this in 3 hours." Conveyor needs all of those — per-job rate limiting and 8-hour webhook retries — and builds them out of two humble primitives: the visibility timeout and DelaySeconds. Once you see this, the "advanced patterns" stop being advanced. They're just these two levers, used well.

The reframe: the borrow clock is writable

In Lesson 1 the visibility timeout was something SQS sets when it hands you a message. But you can also set it yourself, on a live in-flight message, with ChangeMessageVisibility. That one call turns "when does this message come back?" into a dial you control — and that dial is enough to build flow control.

Two levers for "later" DelaySeconds — set at send time: "don't make this message visible for N seconds after it's enqueued." Change­Message­Visibility — set after receive: "I've got this message, but hide it again for N seconds instead of deleting it." Same effect (a message disappears and reappears later); different moment. Almost every SQS timing trick is one of these two.

Pattern 1 · Backpressure by bouncing (the visibility lever)

Conveyor caps how many tasks a single job runs at once. When a worker picks up a task for a job that's already at its ceiling, it does not fail and it does not process — it returns a sentinel, ErrThrottled ("not a failure; the limiter already counted the throttle"). The dispatcher catches that and bounces the message: 

// dispatcher.go — the throttle branch
case errors.Is(err, concurrency.ErrThrottled):
    d.markThrottled(jobID)                    // 1. cool the queue down
    d.bounce(ctx, q, m, adaptiveBounceVisibility(m.ReceiveCount))
    // bounce = q.ChangeVisibility(m, visibility) — hide it, try later

No delay queue, no Redis, no cron. "Try this again in a bit" is expressed purely by re-hiding the message. When a permit frees up, the message reappears and some worker runs it. That is backpressure built entirely from the borrow clock.

Why the bounce backs off exponentially The bounce visibility isn't constant — it doubles per receive: 5s → 10s → 20s → 40s → 80s → 160s → 300s (capped). Two reasons, and the second is the subtle one. (1) A persistently-throttled job backs off more, like any good flow control. (2) Remember Lesson 4: every bounce is a receive, eating the maxReceiveCount: 20 budget. Exponential spacing means a waiting task can't be re-received fast enough to burn through 20 and get wrongly dead-lettered. The backoff curve and the DLQ threshold were designed together.
Three cooperating layers — read them in the code Bouncing is the last resort, not the only move. Conveyor stacks three: (1) pre-poll gate — pickRandom skips a queue whose local in-flight already hit the limit, so you don't even receive work you can't run. (2) cooldown — markThrottled makes the pool skip that queue for ~5s, so 100 goroutines don't all bounce it at once. (3) bounce — for whatever still slipped through, re-hide it with the exponential visibility. Avoid receiving → avoid re-picking → defer the remainder. That layering is the craft.

Pattern 2 · Long delays by multi-hop resend (the DelaySeconds lever)

Webhook delivery retries on a curve: 30s → 8h with jitter, over a 3-day budget (webhook.go). But there's a wall: SQS caps DelaySeconds at 15 minutes. How do you wait 8 hours with a 15-minute primitive? You hop.

1
Attempt fails — compute next interval, say 2h. Stamp NextAttemptAt = now + 2h into the message body.
2
Resend with delay = 15m (the cap). The message vanishes for 15 minutes.
3
It reappears, handler re-checks — is now past NextAttemptAt? Not yet (only 15m passed). So re-defer: resend with another 15m delay. Attempt is not incremented — this hop made no real attempt.
4
…repeat ~8 hops… until now ≥ NextAttemptAt, then actually POST the webhook. A long timer, assembled from short ones.

The full state lives in the message body (Attempt, NextAttemptAt, LastError), so each hop is self-contained — no timer table, no scheduler row. The queue is the timer.

The one idea behind both patterns

Say it back to yourself SQS has no priority lanes, no scheduler, no native long delays. When you need those, you don't reach for another system — you synthesise timing out of visibility and delay. Backpressure is "hide it and re-show it later"; a scheduled retry is "hide it repeatedly until the clock catches up." That move — turning a delivery primitive into a control plane — is what separates someone who uses SQS from someone who designs on it. You're now the second kind.
SQS

Per-message visibility + DelaySeconds. A surprisingly strong primitive — you built rate limiting and an 8h retry timer with nothing else.

NATS JetStream

Has this natively: AckNak with a delay is exactly the bounce — "redeliver this in N seconds." Conveyor hand-rolls what JetStream ships.

Kafka

No per-message delay at all. To back off you pause the partition or seek — blunt instruments. Visibility-style per-message control is genuinely something Kafka can't do.

Read this next

Primary sources: AWS · Changing a message's visibility timeout and AWS · Delay queues (DelaySeconds). Then read, in order: the throttle branch of processOnce and adaptiveBounceVisibility in dispatcher.go, then deferDelivery + scheduleRetryOrGiveUp in webhook.go.

This is the densest lesson so far — if the bounce/cooldown/gate interplay or the multi-hop timer is fuzzy, ask me to trace a single throttled task or a single 2-hour retry step by step. That's the best way to lock it in.

← Lesson 4 · DLQs Next → The dynamic queue-per-run architecture