Evidence: 5 file:line witnesses as of 2026-08-18. Confidence: partial, some rules witnessed, the rest inherited from the seams. What changes it: the first module built against this doc (../PLAN.md forge-1) and its instance breakdown.

What a module writes down about what it did, for whom: per-job stats, the per-item log a caller can read, tracing, and cost. In versable-runner this is the largest concern in the codebase, larger than storage and auth together (app/observability.py + app/usage.py, 462 lines), which is not what anyone would guess and is the right instinct. What is missing everywhere is the thing the owner named first: a caller debugging one item out of 5,000, from another cloud, without shell access to the module.

Audience: anyone building a module's stats, logs, or traces, or an app that needs to show a customer what happened.

What the instances do#

The rule#

Every job has stats a caller can read without a shell; every item has a log a caller can read without a shell; every LLM call is traced; and every number carries the correlation ids.

Per-job stats#

The /jobs/{job_id}/stats shape versable-runner already serves is the contract, because it was built by watching real jobs and it answers the questions people ask: how far along, how fast, when will it finish, what failed and why, what did it cost. Kept as-is, plus tenant and attribution on the record, plus by_variant and by_judge breakdowns once variants exist (14-graceful-degradation.md).

/stats (module-wide) is filtered by tenant by default, because a caller sees its own tenants' work and nothing else; a module operator's cross-tenant view is a scope (stats:admin), not the default.

The per-item log#

This is the missing piece, and it is a contract requirement because the owner named it: apps call modules "usually indirectly or directly via a human OR a workflow run ... and [need to] have logs / re-runs / detailed data inspection", and "it works on my computer but not on prod" is the recurring friction. A caller must be able to fetch, for one item:

• every attempt, with its start, end, and outcome
• what the payload did in each attempt, at a level a person can follow: which template, which research query, which vendor call, what came back, what was retried and why
• the outcome and the usage for that item

Served as GET /jobs/{job_id}/outcomes/{item_id}/log, paginated, plain structured lines (ts, level, attempt, message, optional data), written by the one logger the runner hands the payload (01-runner-and- payload.md, the verb set), scoped to job and item automatically so the payload never writes an id.

Storage: append-only lines under the item's prefix, chunked the way speedway does it (joblog.server.ts:16-97, about 40 lines per doc, token-suffixed chunk ids so two workers on a recovered run append instead of clobber, read sorted by each line's own timestamp). walmart's 20-line, latest-run-only stage log with prior rows deleted on write is the shape to avoid: it answers "what happened last time" and nothing about the time that failed.

Verbosity is a per-job setting (settings.log_level in the job envelope, default info), and a debug re-run of one item is a single-item child job with log_level: debug and parent_job_id set, which is the "re-run and inspect" path an app needs without shell access.

Tracing#

Langfuse is the tracing target the estate already uses in two of three instances, and it stays: every LLM call inside a payload is wrapped so a trace exists per item attempt, tagged with tenant, job, item, capability, variant, and model. Off by default (keys empty means no-op, which is how both instances behave), on in dev and prod. It is not the caller-facing log; it is the operator's view of the model calls, and its ids are recorded on the item log so a person can jump from one to the other. The wrapper is the runner's verb (01-runner-and-payload.md); a payload that imports the tracing client itself, as versable-runner's lib/providers/gemini does with Langfuse, has taken an observability decision the runner should own, and it is why services-api has tracing configured in lib/config and nowhere in app/.

Error tracking#

Unhandled exceptions and runner faults go to an error tracker (Sentry is what App V5 uses, centralized to one init_sentry after "prior code had four separate sentry_sdk.init(...)", lib/sentry/__init__.py), initialized once by the runner at boot, tagged with the same correlation ids, off without a DSN. It is distinct from the per-item log (what happened to an item) and from tracing (what a model call did): it is what broke in the runner. A payload never initializes it.

Health and build identity#

/health (liveness), /health/deep (storage and queue round-trip, effective non-secret config, verifier in use, reference-data versions), /build-info ({commit, built_at, branch?, name, version, contract_version}, snake_case, the same shape an app serves): required routes, contracts/module-surface.md. Two traps worth writing here because two instances hit them: /healthz is edge-reserved on *.run.app and never reaches the container (speedway server.js:14-16 names it), and a health route that only says ok cannot tell you the bucket is gone.

Cost is observability#

Duration, attempts, tokens, LLM cost by model, and vendor call counts are recorded per item as usage events (08-usage-and-credits.md) and rolled up in stats. The point for this doc: cost is a first-class number a caller can read per job and per item, not something reconstructed from a cloud bill later. versable-runner's per-model rollup (observability.py, the services-api additions) is the shape.

Correlation, again#

Every stat, log line, trace, and usage event carries tenant, caller, job, item, attempt, capability, variant, and the attribution keys. This is stated in 03-jobs-and-state.md and repeated here because observability is where its absence hurts: a per-stage success rate for one customer's workflow run is a filter, not a project, if the ids are there, and impossible if they are not.

Do-nots#

• Do not cap a per-item log at the last run. Append, chunk, and keep every attempt. (walmart orchestrator.py:255-264)
• Do not require a shell to answer "why did item 4,312 fail". (all three instances today)
• Do not let a payload write its own ids into log lines. The runner scopes the logger.
• Do not name a Cloud Run health route /healthz. (speedway server.js:14)
• Do not serve a health route that cannot go red.
• Do not emit a stat, log, trace, or usage event without tenant and job on it.