Evidence: 22 file:line witnesses as of 2026-08-18. Confidence: high, most rules here have a named witness. What changes it: the first module built against this doc (../PLAN.md forge-1) and its instance breakdown.

How accepted work becomes running work and how it survives everything that goes wrong on the way: the queue, concurrency, retries, dead workers, cancellation, restarts, duplicate delivery, and slow vendors. This is where the instances have each solved part of the problem and none has solved all of it, so the contract is the union of what worked, stated as the runner's obligations.

Audience: anyone writing or reviewing a module's runner.

Where each instance stands#

The current walmart pipeline can be left running forever by a dead worker and can be deleted mid-run (../evidence/20260817-walmart-recon.md §5). That is the shape of what happens when these are optional. It is also a regression: the legacy Catalog pipeline it replaced had both heartbeat and cancel (backend/app/dispatch.py), and nothing caught the loss because no artifact represented the old behaviour. A rewrite of a runner carries a parity ledger against what it replaces, row by row of this table, before the old one is removed.

The queue is a port#

A module talks to its queue through one interface with two obligations: enqueue a unit of work with a deterministic name, and deliver it to a handler at least once. Everything else the queue offers (its own retry, its own concurrency, its own deadlines) is a backstop the module configures but does not rely on for correctness.

Three adapters, selected by config, all witnessed:

The adapter must be swappable without touching the payload or the job surface. That is the test of the seam.

Submit is O(1) in items#

Creating a job persists the envelope and the items and enqueues one unit of work, the expansion. The expansion turns items into per-item units and is itself idempotent (deterministic names, so a redelivered expansion re-creates nothing). A 5,000-item job returns its id in the time it takes to write two records. Until expansion finishes the job is expanding, and that is a state callers see (03-jobs-and-state.md). versable-runner does exactly this (app/jobs.py:93-106, run_fanout at :110); the reason recorded in the code is that Cloud Run throttles CPU outside a request, so fanning out inside a task's request is the only place it runs at full speed.

Name the semaphore#

Every module states, in its manifest and its config, what bounds concurrent work and at what scope:

• the queue-level ceilings, all of them, not only the headline one: Cloud Tasks has maxConcurrentDispatches, maxDispatchesPerSecond, maxAttempts, maxBurstSize, and the backoff bounds, and versable-runner's deploy.sh:39-42 sets five; arq has max_jobs; in-process has the semaphore size
• the module-level ceiling per caller and per tenant, enforced at admission, so one caller cannot take the whole queue
• outbound ceilings per vendor (Oxylabs, Gemini, OpenAI), shared across workers when there is more than one; walmart's Redis-backed slot limiter (backend/app/ratelimit.py) is the shape once workers scale past a single process, and env caps like speedway's OXYLABS_CONCURRENCY are enough before that

Where the number lives matters as much as what it is: App V5 sets worker concurrency as a deployment count (25 instances in prod, 5 in dev, at the Render service level), which is easy to change and hard to discover from the code or /health/deep. The rule: every ceiling is readable from the module (config echo, manifest ceilings), wherever it is set.

A ceiling is also shared by whoever points at it. services-api and runner-service are both configured onto the pipeline-runner queue, so two independently deployed services draw on one maxConcurrentDispatches and neither can be throttled without throttling the other. A queue per service, or per environment, is the default; sharing one is a decision to state, not a thing to discover.

versable-runner's single queue ceiling is elegant and it means two tenants share one throughput budget with no fairness. Two answers, both allowed: fairness at the app (the app decides what to submit when), or fairness at claim time in the runner, which App V5 does by picking a random job first and then a task within it (lib/tasks/claimer.py:80-81), so a 300k-item job cannot starve a 50-item one. A pull-queue runner can do the second; a push queue (Cloud Tasks) cannot, and gets fairness from per-caller admission plus app-side scheduling. Either way the module makes its ceilings explicit and per-caller admission real.

The handler owns the retry budget#

The unit-of-work handler decides whether a failure is retryable, counts attempts, backs off, and on exhaustion writes the terminal error outcome and returns success to the queue so the queue stops. The queue's own retry is configured higher and only fires if the handler dies without answering. Both runner forks do this and document it (docs/runner-service.md, retry semantics in ../evidence/20260817-runner-architecture.md); the prior session called the mismatch between the two numbers a bug and had to retract it, which is why the reason is written here.

• Payloads raise a non-retryable error type for "this will never work" (bad input, verification failed, vendor said no permanently). The runner writes the outcome on the first attempt.
• Everything else is retried up to the job's max_attempts with backoff. Flat backoff is fine at small attempt counts; exponential with jitter once attempts exceed three or the vendor is rate-limiting.
• The attempt count is recorded on the outcome.

Anything longer than one dispatch heartbeats#

A unit of work that runs longer than the queue's delivery deadline, or that loops over many items inside one delivery, stamps a heartbeat as it goes, and the runner treats a stale heartbeat as a dead worker: it re-queues the unit from its checkpoint, up to a recovery cap, then marks it errored for a human. speedway is the witness (app/lib/runs.server.ts:231-353, STALL_MS 3 min, MAX_RECOVERIES 3). One item per delivery, as versable-runner does, needs no heartbeat because the queue's redelivery is the recovery; the moment a payload batches, it needs one.

Recovery is transactional and idempotent: re-queueing a unit that has since finished must be a no-op, which is what first-writer-wins finalization gives you. When the fast heartbeat store can itself be down, a slow backstop sweep on the durable store catches what the fast path missed: App V5's Redis 15 s TTL plus a 30-min Mongo lock-timeout sweep (lib/redis/worker_heartbeat.py:8-9, lib/tasks/claimer.py:404-449) is the shape.

Checkpointing is the runner's helper, not the payload's habit#

A payload that iterates over many items receives an iterator from the runner: give me the next batch; here is my progress. The runner persists the cursor and the heartbeat, resumes from the cursor after a restart or a recovery, and never asks the payload to remember where it was. speedway's three near-identical copies of that loop (partType.server.ts:114-187, normalize.server.ts:440-501, content.server.ts:628-733) are the cost of not having it.

Cancel is cooperative, and its end state is defined#

POST /jobs/{job_id}/cancel sets a cancel request, deletes or skips units not yet started, and lets in-flight units finish or check the flag between batches. The job moves to cancelled when nothing is pending or in flight. Items not started get a skipped outcome with reason cancelled; items in flight when the flag was set finish and record their real outcome. A payload that calls a vendor with its own job lifecycle gets a hook to cancel vendor-side, best effort (speedway's abortExtractorRunDispatches, app/lib/extractor.server.ts:219-226).

If a run is already stalled when cancel arrives, cancel finalizes it immediately rather than waiting for a worker that will never answer (speedway, runs.server.ts:199-229).

Idempotency is structural#

Every write that costs money or ends something is guarded, not by a transaction around the whole world, but by making duplicates harmless:

• unit names are deterministic ({job_id}/{item_id}/{attempt} or the equivalent), so a re-enqueue is a no-op
• outcome writes are create-if-absent (if_generation_match=0 on GCS, a unique key elsewhere), so a redelivered unit cannot clobber a result
• job finalization is first-writer-wins
• a paid outbound call is preceded by an intent record, so a crash mid-call leaves a traceable hold instead of a silent second charge (speedway, scrape.server.ts:456-475)
• usage events carry an idempotency key and the ledger dedups on it (08-usage-and-credits.md)

The instances arrived at this shape independently (../evidence/20260817-speedway-recon.md §4, "the same write-intent, check-before-acting, terminal-state-guard shape, arrived at independently"), which is a strong sign it is the shape.

Waiting on something slow is a runner primitive#

Anything that has to wait for a vendor job, a feed verdict, or a webhook gets a poll-with-backoff primitive from the runner: schedule a check, back off, cap the total wait, transition to a terminal state on timeout, respect cancel.

walmart's poll_feed_status (backend/app/jobs.py:636-771) rebuilt all of it, and the interesting part is where. jobs.py is a runner file by walmart's own seam (instances/walmart-mvp.md, runner file list), so this is not a payload reaching across the boundary. It is a second implementation of "how long-running background work is retried" sitting beside the first, in the same layer, in one codebase. One is enough, and the seam being intact is no defence: a missing primitive gets rebuilt by whoever needs it, and sometimes that is the runner itself.

Deferring is not failing, and breakers are the runner's#

When a unit cannot proceed because something shared is unhealthy (a vendor breaker is open, a fleet-wide retry budget for that vendor is spent, a rate limit says wait), the payload signals deferred and the runner re-queues the unit later without counting an attempt, up to a deferral ceiling, after which it is an error (03-jobs-and-state.md). App V5 has the whole mechanism: a per-process sliding-window breaker with a fleet-consensus layer over Redis, a Redis retry budget per worker and downstream with a 60 s window that fails open, and a decorator that fixes breaker-outermost, retry-innermost so callers cannot invert it (lib/breakers/*, lib/redis/retry_budget.py, lib/breakers/decorator.py:1-11). Two lessons carry: the breaker and the budget are runner primitives the payload reaches through Context (../contracts/runner-verbs.md), never its own imports; and "deferred" is a distinct disposition so a healthy item is not marked failed because a vendor had a bad minute.

Timeouts sit under the platform's#

Per-unit timeouts are set below the queue's dispatch deadline and the platform's request timeout, so the handler gives up and records the outcome before the platform kills the container and records nothing (versable-runner: RUNNER_ITEM_TIMEOUT_S = 1700 under Cloud Tasks' 1800). Memory is part of the same budget: walmart dropped concurrency from 5 to 2 after an OOM on a 512 Mi worker (worker.py:116-121), and the reason is in the code, which is the right place for it.

Dead letters#

A unit that exhausts recovery, or a callback or usage event that cannot be delivered after retries, goes to a dead-letter record with everything needed to replay it, and a human-visible count. speedway's usage engine has one with replay pinned to the original billing period (app/lib/usage/usage.server.ts). Silent drops are how "it says succeeded but the customer was never charged" happens.

Do-nots#

• Do not rely on the queue's retry for correctness. The handler owns the budget; the queue is a backstop. (both runner forks, documented)
• Do not batch inside one delivery without a heartbeat and a checkpoint. (walmart's current pipeline)
• Do not ship a job surface without cancel, or let a running job be deleted. (walmart routes/jobs.py:215-216)
• Do not put checkpoint, backoff, or poll machinery inside a payload. (speedway partType.server.ts:114, normalize.server.ts:440, content.server.ts:628)
• Do not build a second one inside the runner either. (walmart jobs.py:636-771, beside orchestrator.py)
• Do not enqueue N units at submit time. Enqueue one expansion. (versable-runner jobs.py:93)
• Do not write an outcome or charge a vendor without an idempotency guard.
• Do not let a per-unit timeout exceed the platform's.
• Do not drop an undeliverable event silently. Dead-letter it and count it.