Agent docs

Jobs and state

> Evidence: 9 file:line witnesses as of 2026-08-18.

Evidence: 9 file:line witnesses as of 2026-08-18. Confidence: partial, some rules witnessed, the rest inherited from the seams. What changes it: the first module built against this doc (../PLAN.md forge-1) and its instance breakdown.

What a job is, which states it can be in, what happened to each item, who owns that state, how a caller reads it, and how a caller learns it changed. The instances split on storage and agree on almost everything observable, so the contract fixes the observable part and leaves storage to a recommendation with the condition attached.

Audience: anyone building a module's job surface, or an app that reads it.

The job#

A job is one capability over N items, from one caller, for one tenant. It is created in one call, gets an id immediately, and from then on is addressable by that id for status, results, errors, stats, and cancel. Nothing about a job depends on the caller staying connected.

The envelope every job carries, from creation to retention:

FieldSet byMeaning
job_idmodulestable id; the module's own
client_job_idcaller, optionalthe caller's id for this work; unique per caller; used for idempotent create and for the caller-owned state mode below
capabilitycallerwhich capability of this module to run; must be in the manifest or the create fails fast
paramscallercapability parameters, validated against the manifest schema
itemscallerthe N inputs, each with an item_id (caller-supplied or positional) and an input object (called payload on the wire in versable-runner; renamed so "payload" means only the capability code, 01-system-classes.md)
tenantrequestinherited from the request, never from the body alone
caller_id, envverifierfrom the caller context
attributioncaller, optionalclaimed context: user, workflow run, agent; stored verbatim, searchable. attribution.user is reserved with the shape {id, label} so an app can render and facet "who ran it" without guessing; the manifest names which attribution keys are facetable; the rest is free
created_at, updated_atmoduleupdated_at moves on every state or count change, so a running list can sort by it
settingscaller, optionalper-job overrides the manifest allows: max attempts, per-item timeout, callback URL

versable-runner already has all of this except client_job_id, tenant, and caller_id, and it calls attribution meta (src/services-api/app/api.py, docs/runner-service.md). walmart-mvp and speedway have the equivalents inside their own job models.

Job states#

accepted ──► expanding ──► running ──┬──► succeeded
├──► completed_with_errors
├──► completed_with_review
├──► cancelled
└──► failed
StateMeaningWho can see it
acceptedpersisted, id returned, nothing dispatched yetevery caller, briefly
expandingthe module is turning the items into work; a big job sits here for real secondsevery caller; versable-runner calls it enqueuing and it is a genuinely reachable state
runningat least one item is in flight or waiting
succeededevery item has a resultterminal
completed_with_errorsevery item is settled, at least one is a terminal errorterminal
completed_with_reviewevery item is settled, none errored, at least one needs reviewterminal from the module's point of view; the app may re-submit resolved items as a new job
cancelleda cancel was requested and every in-flight item has finished; nothing pending remainsterminal
failedthe job itself could not run: expansion failed permanently, params invalid after acceptance, storage goneterminal, rare, always carries a reason

Two optional behaviours a module may add. They are different kinds of thing and they are declared on different surfaces, so the manifest names each in its own place (contracts/manifest.md).

paused is a job state: pending units held, in-flight units finish, resumable to running (App V5 has it at job and item level, api/jobs.py:150-170; speedway holds dispatch at create with setupHold). A module implementing it lists it in the manifest's optional states.

deferred is a per-unit disposition and NOT a state, and not an outcome either: the unit is re-queued later without burning an attempt because a dependency was unavailable (a breaker open, a vendor rate limit; App V5 task_runner.py:341-350). Because it is not a state it never appears in states. A module using it declares the bound as ceilings.max_deferrals, and past that bound the unit becomes an error outcome with DEFERRED_TOO_LONG, which therefore has to appear in that capability's error_types like any other emitted code.

Two rules about the set:

  • accepted and expanding are real. Submit is O(1) in item count (04-dispatch-and-workers.md), so a caller that reads status right after create will see one of them. A caller that cannot tolerate "accepted but not yet running" needs to say so and wait, not be lied to with running.
  • Terminal means terminal. No transition leaves a terminal state. A resolved review or a retry of failed items is a new job that references the old one (parent_job_id), which keeps every job's history immutable and every stat honest. speedway's settleRunOnce (first terminal write wins, app/lib/runs.server.ts:90-104) is the guard that makes this true under duplicate delivery.

Alongside the state, a job always reports counts: total, pending, running, succeeded, errored, needs_review, cancelled, skipped. The counts are the truth; the state is derived from them (below).

Item outcomes#

Every item ends in exactly one outcome, written once:

OutcomeCarriesTerminal?
resultthe payload's output, plus confidence (0 to 1) and evidence where the capability has oneyes
errorerror_type (a stable code from the manifest), message, retryable: false, attemptsyes
needs_reviewreason (a stable code), confidence, evidence, and the payload's best partial output if anyyes, for the module
skippedreason; the item was excluded before work (duplicate, filtered, cancelled before start)yes

And every outcome carries: item_id, attempts, duration_ms, usage_ref (an array of the usage event ids for this item, one per meter per attempt, contracts/usage-event.md), finished_at. Correlation is job_id, item_id and attempts; nothing else is stamped for it. Conditionally, and named so no two modules invent them: variant (an object of dimension to option, when the capability declares variants), reference_versions (an array of {name, version}, when it declares reference_data), and on result and needs_review judged_by (self or the judge's capability id) with confidence (contracts/module-surface.md shows all of them).

Confidence is a number with a source. confidence is 0 to 1; judged_by says where it came from: self (the payload rated its own output), or the id of a judge capability that scored it (a second model call inside the module, a separate judge module, or a judge the app ran and wrote back). Owner ruling 2026-08-17: self-report is the norm and is not good enough in some cases, so the contract does not lock into it; a judge is a capability variant (14-graceful-degradation.md) that can sit in any of those three places, and the app applies the customer's bar over whichever it gets. When more than one score exists for an item, the module records all of them and the outcome's confidence is the one its manifest says is authoritative.

needs_review is how a module says "I could not settle this and a person should look" without guessing and without dropping. It is a per-item outcome, not a job state, and the queue that holds it belongs to the app (10-human-in-the-loop.md). Confidence rides on result too, because the app applies the customer's bar on top and may hold an item the module called done (../evidence/20260817-source-docs-skeptical-read.md §C).

retryable: false is written on purpose. A retryable failure is not an outcome; it is an attempt. Only the terminal one is recorded as error, and the attempt count says how many tries it took. versable-runner writes errors/{idx}.json on exhaustion and returns success to the queue so it stops (docs/runner-service.md "How a job runs" step 4).

Who owns the state#

Three modes, and a module supports the first and must accept the ids that make the second and third possible.

Module-owned (default). The module is the source of truth for job and item state. The caller reads it (poll or callback) and mirrors what it needs for its own UI. versable-runner.

Caller-owned, module-mirrored. The caller already has a job and item records (a workflow run, a Firestore job doc, a Postgres Job row) and wants the module to work against its ids. The caller passes client_job_id and per-item item_id; the module stores its own outcomes keyed by them; the caller's store stays the truth for its UI and the module's store stays the truth for module-side outcomes. Duplication is accepted and the id is how the two are joined. This is what the owner described on 2026-08-17 ("the caller provides a state and the module merely mirrors it, can duplicate somewhat and let the id be the resolution") and it is how walmart-mvp and speedway would consume a module without giving up their job models.

In-process. The module runs inside the caller's process (13-local-dev-and-debugging.md); state lives wherever the caller's runner puts it, and the module's runner is not deployed at all. Same outcome shapes, no wire.

What all three share: client_job_id + item_id are always accepted and always echoed back, creating a job with a client_job_id the module has already seen from this caller returns the existing job (idempotent create), and outcomes are always addressable by item_id.

Derived or stored#

versable-runner derives job state from blob listings (manifest count against results/, errors/, usage/), so there is no status column to go stale and a crashed process leaves no lock (../evidence/20260817-runner-architecture.md). speedway derives job status inside a transaction from the stages map and stores run status (app/lib/jobs.server.ts:245-311). walmart-mvp stores everything and funnels every job-level mutation through one function, evaluate_job (backend/app/orchestrator.py:507-556).

The contract does not pick a storage. It fixes what a caller sees (states, counts, transitions, outcomes above) and recommends:

  • Derive when outcomes are individually addressable and listing them is cheap. One outcome record per item, in a store that lists by prefix or index quickly, and the state is a count. Nothing to corrupt. This stops being viable when a caller needs to query jobs by a field that is not in the key, or when the item count makes a listing a scan (versable-runner's own condition, ../evidence/20260817-runner-blueprint-approach.md).
  • Otherwise store, with one mutation point. A single function owns every job-level transition; no other code writes the state field. Terminal writes are first-writer-wins.

Either way, the counts and the state a caller reads are the same, and a module may switch storage without changing its surface.

Reading a job#

Every module exposes, for every job:

  • status and counts, cheap enough to poll every few seconds
  • outcomes, paginated (offset/limit or a cursor), filterable by outcome type, with a total; only the requested page is loaded (versable-runner's ?offset=&limit= reads only the page's blobs, docs/runner-service.md)
  • one item's outcome by item_id
  • errors and review items as their own lists, since they are what a human looks at
  • stats: timing (first and last outcome, elapsed, throughput, ETA while running), error breakdown by type with samples, usage rollup, cost
  • the envelope the job was created with, verbatim: this is the app's "settings snapshot" view of what was frozen at consent, so a page can render it without a second source
  • the job list, newest first, filterable by state, capability, tenant, created window, and attribution keys, paginated

Every list in a module is paginated. There is no unbounded list anywhere.

Learning that something changed#

Polling is the floor, callbacks are the target, and a sweep is the guard. The extractor's history is the reason this is a rule, though the history is more specific than it first looked: polling was the caller's reported pain ("frequent polling for job status", owner, 2026-08-17), and the extractor also delivers webhooks. The surfaces differ by deployment. Speedway's client receives signed callbacks and runs a reconciliation sweep (speedway/app/lib/extractor.server.ts:5-9, speedway/app/routes/tasks/extractor-webhook.tsx:20-21); the automation surface in instances/extraction.md offers poll only.

So the lesson is not "that module forgot callbacks". It is that a capability present on one deployment and absent on another leaves every caller building for the weakest one, which is what a contract exists to stop.

  • Poll. Status is always readable and always cheap. Every caller can live on this alone.
  • Callback. A job may carry a callback_url. The module POSTs a signed event on every job state transition and, if asked, on batches of item outcomes. Delivery is at-least-once with retries and backoff; the receiver must be idempotent on (job_id, event_id). Signature is HMAC over the body with a per-caller secret the issuer hands out, so the receiver can trust the sender.
  • Sweep. The caller runs a periodic reconciliation that lists jobs it believes are open and reads their status, so a lost callback is a delay, never a stuck job. The module's job list with a state filter is what makes this cheap.

Correlation#

Every job carries tenant, caller_id, env, and the attribution keys (user, workflow run, whatever the app sends), and every outcome, usage event, and log line inherits them. This is what makes a per-SKU-through-workflow price and a per-stage success rate computable from module records (../evidence/20260817-source-docs-skeptical-read.md §D and §H). A module that drops the correlation on the way to storage has broken the contract even if every job succeeds.

Do-nots#

  • Do not report running for a job that has not been expanded yet. Report accepted or expanding and let the caller wait.
  • Do not leave a terminal state. Resolution and retry are new jobs with a parent_job_id.
  • Do not write a retryable failure as an outcome. Only the terminal attempt is an error.
  • Do not expose an unpaginated list.
  • Do not require the caller to poll. Offer a callback and document the sweep. (extractor, per owner; speedway had to build around it)
  • Do not accept a job without a tenant, and do not let a caller's meta or attribution stand in for one. (versable-runner)
  • Do not let a job be deleted or mutated while it is running. (walmart-mvp routes/jobs.py:215-216, the commented-out guard)
  • Do not put "stage" on the module surface. A stage is the app's composition over capabilities and child jobs; the module gives capability and parent_job_id and the app reconstructs the stage. (vb-fable's shape check against both shipped jobs tables, 2026-08-18)
  • Do not let a read mutate. A GET on outcomes is idempotent and safe to poll; "first pull wins" belongs on an explicit acknowledge call, never on a read. (the extractor's /jobs/{job_id}/results marks the job consumed on first pull, speedway/app/lib/extractor.server.ts:11-12)
@versable-git/ui · reference, canon, and method, read in place