Queues · Lesson 1

The shape of SQS: a pull-based buffer with a borrow clock

You read ZenRows' SQS code and want to know, not guess, what happens to a scraping job on a slow worker, a crashed worker, or a retry. Every one of those answers comes from this one lesson. Master this and the rest of the queue is just elaboration.

One idea: a message is borrowed, never just "read"

The instinct from training data is "a queue is a pipe — you push in one end, it comes out the other." SQS is not that. SQS is a durable buffer that consumers pull from, and the single mechanic that defines its whole personality is this:

The core rule Receiving a message does not remove it. It only hides it for a while. The message is gone for good only when the consumer makes a separate DeleteMessage call. If that call never comes, the message comes back.

That "for a while" is the visibility timeout — think of it as a borrow clock. When a worker receives a message, SQS lends it out and starts a timer. Until the worker deletes it or the timer runs out, no other worker can see it. Here is the full lifecycle for a ZenRows scrape job:

1
Produce — the crawler calls SendMessage with {"url": "https://target/page/42"}. SQS stores it redundantly across servers. It is now visible.
2
Receive — a scraping worker calls ReceiveMessage and gets the job. The borrow clock starts. The message is now "in flight" — invisible to every other worker.
3
Process — the worker fetches the page, solves the challenge, parses it. The message still sits in the queue, hidden, the whole time.
4
Delete — on success the worker calls DeleteMessage. Only now is the job truly gone. No delete, no removal.

Notice what's missing: SQS never pushed anything. Workers ask for work when they have capacity. That's why SQS scales by "just add more workers" — the queue is a buffer that absorbs the gap between how fast jobs arrive and how fast you can scrape them.

Why scraping makes the borrow clock interesting

Scraping durations are wildly variable — a static page is 200 ms, a JS-heavy page behind a CAPTCHA might be 90 seconds. The visibility timeout is your bet on "how long should a worker get before I assume it died?" Two ways to lose that bet:

Failure mode · timeout too short Timeout is 30 s, the scrape takes 90 s. At second 30 SQS decides the worker is dead and hands the same job to a second worker. Now two workers scrape the same URL, burn two proxies, and both try to delete it → duplicate processing. Fixes: raise the timeout, or have the worker periodically extend its own clock with ChangeMessageVisibility (a "heartbeat").
Failure mode · worker crashes — this one is a feature A worker pulls a job, then its container is OOM-killed mid-scrape. It never deletes the message. At the timeout, the job automatically reappears and another worker picks it up. You didn't lose the URL. This is SQS's core promise: at-least-once delivery — nothing is dropped just because a consumer died.

"At least once" is a double-edged phrase, and it's the seed of your next lesson: if a job can be delivered more than once (slow worker and crash recovery both cause it), then your scrape-and-store logic must be safe to run twice on the same URL. That property is called idempotency, and it's the price of admission for SQS standard queues.

In conveyor · your actual code The per-run task queue (conveyor-{env}-run-{run_id}, a Standard queue) is created with a visibility timeout of a flat 5 minutes (internal/queue/sqs/perjob.go — note the doc comment claims "6× the gateway timeout" but the constructor hard-codes 5m; read the code, not the comment — that's the whole reason you're here). The shared control queue uses max(900s, 6× the Lambda timeout) (infra/lambda/sqs.tf). Both comfortably exceed the work they cover — the same bet as above. And conveyor deliberately does not heartbeat: if a task overruns and gets redelivered, that's fine, because the result write is idempotent and the worker checks task status first (Lesson 2). So it chose "generous timeout + idempotent processing" over "tight timeout + heartbeat." (It does call ChangeMessageVisibility — for backpressure, not heartbeating. That's Lesson 5.)

Where this sits among the other brokers

You'll understand SQS better by seeing what it deliberately didn't do. Same mechanic — "don't lose a message if the consumer dies" — solved three ways:

SQS

Per-message borrow clock. Ack = delete the message. The broker tracks each message's state. Simple; no ordering across the queue by default.

RabbitMQ

Also per-message acks, but the broker pushes to consumers and holds an unacked message on that consumer's channel until ack/nack or disconnect.

Kafka

No per-message clock at all. The log keeps every message; the consumer just records an offset ("I've read up to here"). Redelivery = rewind the offset. Different universe — we'll contrast it properly later.

Read this next

Primary source — the one page worth reading in full after this lesson: AWS · Amazon SQS visibility timeout. Then skim Basic SQS architecture for the lifecycle diagram in AWS's own words.

Stuck on anything — why two-step delete, what counts as "in flight," how heartbeating works in code? Ask me. I'm your teacher for this; that's what I'm here for.

Lesson 1 of the course Next → At-least-once & idempotency: surviving duplicates