Queues · Lesson 2

At-least-once: stop fighting duplicates, make them boring

Lesson 1 left you with a loaded fact: a slow worker or a crashed worker both cause the same job to be delivered again. So HandleMessage in your task.go can run twice for the same task. This lesson is how conveyor makes that a non-event — and it's almost entirely a tour of code you already own.

The reframe: you can't prevent it, so neutralise it

The whole idea On a Standard queue, redelivery is guaranteed to be possible — there is no setting that turns it off. So the goal is never "stop the duplicate." The goal is idempotency: make the second run produce the same end state as the first, with no extra side effects. A job that runs twice should be indistinguishable from one that ran once.

Conveyor neutralises duplicates with two complementary moves. Learn to spot both — almost every idempotent system is some mix of them.

Move 1 · Guard — "have I already finished this?"

Before doing any work, ask whether it's already done. In internal/worker/task/task.go, the first thing the handler does after loading the task is branch on its stored status:

switch task.Status {
case model.TaskStatusPending:
    // Normal first delivery.
case model.TaskStatusProcessing:
    // Redelivery — at-least-once. Result key is deterministic;
    // only one terminal write can win.
case model.TaskStatusSuccessful, model.TaskStatusFailed:
    // Already terminal — nothing to do.
    return nil            // ← settle & delete, do NOT re-scrape
}

That return nil on a terminal status is the guard. The task is already finished, so the handler does no work and tells SQS to delete the message. A redelivered "already done" job costs one DynamoDB read and nothing else — no second scrape, no double proxy spend.

Why a guard alone isn't enough The guard only protects against duplicates that arrive after the first run finished. But two workers can hold the same task at the same time (visibility overrun in Lesson 1: the clock expires while worker A is still scraping, so worker B picks it up too). Both see status processing, both scrape. The guard can't help here — so conveyor needs a second move.

Move 2 · Converge — make concurrent work land in one place

Conveyor stores each result in S3 under a key derived deterministically from (job_id, run_id, task_id) — the same task always computes the same key. So even if workers A and B both scrape the URL, they write to the identical S3 object. Last write wins, the bytes are the same, and the stored result is correct either way. The handler's own comment says it: "Result key is deterministic; only one terminal write can win."

Guard vs converge Guard = skip the work if it's already done (cheap, but can't stop simultaneous duplicates). Converge = let the work happen twice but force both runs to the same destination (handles simultaneity, but pays for the wasted work). Conveyor uses both: guard to skip finished tasks, converge so an in-flight double-scrape can't corrupt state. That layering is the actual skill — not one trick, but knowing which gap each move leaves.

The quiet third move: the ack contract

Remember from Lesson 1 that deleting a message is a separate, explicit act. Conveyor expresses it through the handler's return value, and the choice is itself an idempotency decision:

Situation on delivery	Handler returns	SQS outcome
Task `pending` (first time)	nil after processing	delete — done
Task `processing` (redelivery)	nil after re-processing	delete — converged on same key
Task already `successful`/`failed`	nil immediately	delete — guarded, no work
Malformed JSON body (poison)	nil immediately	delete — don't redeliver garbage
Job or run deleted/terminal	nil immediately	delete — work no longer wanted
DynamoDB read failed (transient)	the error	redeliver — try again later

The non-obvious rule Return nil for permanent outcomes (done, poison, deleted) so you don't burn redeliveries on something that will never succeed. Return an error only for transient failures worth retrying. Get this backwards and a poison message loops until it exhausts maxReceiveCount and lands in the DLQ — which is exactly the machinery of Lesson 4. Returning nil on a real bug also silently drops work. This single decision is where most SQS pain lives.

"But FIFO has exactly-once — doesn't that solve it?"

SQS FIFO

Dedups producer retries within a 5-min window and orders messages. But your consumer can still crash after work, before delete — so you still need idempotent processing. Exactly-once is narrower than it sounds. (Lesson 7.)

Kafka

"Exactly-once" via transactions ties consume+produce+offset into one atomic commit — but only for work that stays inside Kafka. A scrape hitting the outside world is back to at-least-once + idempotency, same as you.

Conveyor's bet

Skip FIFO entirely (all queues Standard), and pay for correctness with guard + converge. Cheaper, higher throughput, and the idempotency was unavoidable anyway. Now you can defend that choice.

Read this next

Primary source: AWS · Standard queues and at-least-once delivery — short, and it states the "design your application to be idempotent" rule in AWS's own words. Then re-read internal/worker/task/task.go lines ~184–205 with this lesson open; it should now read like prose.

Want to go deeper on any of it — how the deterministic key is actually built, what a DynamoDB conditional write (true CAS) would add, or why conveyor tolerates a double-scrape instead of locking? Ask me.

← Lesson 1 · Visibility timeout Next → Long vs short polling: the receive loop