Re-cut of ../evidence/20260817-walmart-recon.md against the canon matrix, plus a Lapses section. Citations are that recon's, read against the working tree at /Users/alcatraz627/Code/Versable/walmart-mvp on 2026-08-17. Live Cloud Run state read the same day.

Two pipeline generations coexist here. Everything below describes the current one (Job to Part to PartError, "pipeline v2"). The legacy Catalog to Upload to UploadItem pipeline appears only where the two diverge, which is exactly where the interesting lapses are: the legacy pipeline has crash recovery and cancel, and the current one does not.

1. What it is#

An app with its runner inside. Python FastAPI serving a SPA, Postgres on Cloud SQL via SQLAlchemy 2 async, Redis with arq for the queue, GCS for bytes, deployed as four Cloud Run services in project walmart-496912: walmart-api, walmart-api-prod, walmart-worker, walmart-worker-prod. One image, two roles, two environments.

Callers are humans in a browser holding a bearer JWT. There is no machine-to-machine inbound path.

The domain is publishing a supplier catalog to Walmart: eight ordered stages from ingest through taxonomy, spec mapping, scrape, images, content, and finally a feed submission to Walmart's API.

2. Shape of the runner#

A single-stage-at-a-time state machine, not a DAG and not a queue of independent tasks. Job.stage (backend/app/models.py:607) names which of eight ordered stages the job is on; Job.stage_state (:608) tracks that stage's own progress.

 trigger (stage task finishes · review row resolved · skip)
        │
        ▼
 evaluate_job()            orchestrator.py:507-556
   the single entry point every trigger calls
        │
        │  gates generically on two per-part structures:
        │    Part.stage_coverage[stage]   did something touch this part
        │    open PartError rows          did it leave a problem
        │
        │  enqueues at most one stage's work, then recurses
        │  synchronously through stages with nothing to do
        ▼
 arq over Redis          deterministic id {task_name}-{job_id}
        │                max_jobs=2 · job_timeout=3600
        ▼
 thin stage wrapper      jobs.py:928-1122
   load job → call one domain module's *_service_v2.py
   → record summary/log → flip stage_state → evaluate_job again

The stage tasks are near-identical wrappers. The seam between "what advances a job" and "what a stage does" is drawn at the module boundary rather than inside a shared function.

3. Concern by concern#

This is the most vendor-entangled instance: Cloud SQL, Redis over a VPC connector, GCS, Walmart's OAuth and feed API, Gemini, Oxylabs, ScraperAPI, and the shared Extractor Service. The runner itself (arq, SQLAlchemy, pydantic) is vendor-neutral; the coupling is concentrated in the payload modules.

4. Runner vs payload#

The seam is clean and drawn at the directory level.

Runner: orchestrator.py, jobs.py, worker.py, dispatch.py, models.py, routes/_common.py, routes/jobs.py, usage.py, security.py, config.py, db.py, storage.py.

Payload: ingest/, taxonomy/, spec/, scraping/, images/, content/, walmart/, plus gemini.py and pcdb/ as shared vendor clients. Each exposes a *_service_v2.py entry point the runner calls by name.

Two named leaks, both from the recon:

1. orchestrator.py carries two lines of Walmart vocabulary: CONTENT_SCHEMA_KEYS (:78) and IMAGE_FIELD_KEYS (:90). The module's own comment (:69-73) says this was deliberate, so neither spec/ nor content/ has to import the other. A documented tradeoff, not an oversight, but the generic orchestrator does know two Walmart facts.

2. poll_feed_status (backend/app/jobs.py:636) builds a second job-lifecycle machine inside the runner: self-re-enqueue with exponential backoff, a time ceiling, an abort flag, terminal-state transitions. The consequence: "how does a long-running background operation get retried or timed out here" has two different answers depending which file you are in.

   An earlier version of this doc attributed that function to walmart/service_v2.py and called it a payload leak. Both halves were wrong. rg -n "def poll_feed_status" over the whole tree returns exactly one hit, in jobs.py, which this doc's own runner list includes; walmart/service_v2.py only calls it. The finding survives the correction but changes shape: this is duplication within the runner, not a payload reaching for something the runner withheld.

Leak 2 still argues for a runner-provided wait primitive, but it is no longer the same failure as speedway's copy-pasted checkpointing (row 13). Speedway's three copies are genuinely in payload files. Speedway remains the only witness in the estate for the payload-leak version of that rule.

5. Deliberate decisions#

Status is stored, not derived, with a single mutation point at the job level. evaluate_job is the only writer of Job.status/stage/stage_state, and orchestrator.py:1-9 names it as such specifically to hold that line. This is the deliberate opposite of speedway and versable-runner, and the tradeoff is stated: always-consistent-by-construction is exchanged for must-be-right-at- every-call-site.

Concurrency is capped low and the reason is on record. arq max_jobs went from 5 to 2 after image_normalization OOM-killed a 512Mi prod worker on 2026-08-11 (worker.py:116-121). A memory-pressure decision, not a throughput default. This is the only instance whose concurrency number has a documented incident behind it.

Submit cost is synchronous and non-trivial. create_job uploads every file and runs the duplicate part-number pre-check inside the request cycle (routes/jobs.py:48-154). Only ingestion is deferred. The opposite of versable-runner's O(1) submit, and a deliberate choice to fail fast on bad input.

A stage running cleanly is not the same as a stage doing something (jobs.py:31-43). matched = processed - flagged silently reads as success for a stage like scrape that never files PartError rows, even when it fetched nothing. matched_override exists as the escape hatch. This is a real bug class someone hit and fixed, and it is the best articulation in the estate of why "no errors" is not a success signal.

Publish is a button, not a stage (routes/jobs.py:176-189), and uniquely requires owner/admin.

Live submission is double-gated: a global settings flag and a per-org DB flag must both be true (config.py:155-159, models.py:46-51), separate again from stub_feed_submit which fakes the whole Walmart round trip for local testing.

Metering is deliberately lossy. usage.py's docstring states a crash can lose one autoflush buffer, "which is fine for dashboard stats". An explicit accuracy-for-simplicity tradeoff.

6. Lapses#

No crash recovery on the current pipeline (11). Canon settles that any run longer than one dispatch needs a heartbeat. Job has no heartbeat column and no retry counter; recover_stale_jobs queries only the legacy Catalog table. The only backstop for a job stuck at stage_state="running" because its worker died is arq's job_timeout=3600. The legacy pipeline this replaced had recovery, so this is a regression, not an unbuilt feature.

No cancel on the current pipeline (12). Canon settles cooperative cancel as mandatory. There is no cancel route and no column. Worse, routes/jobs.py:215-216 carries a commented-out guard that would have blocked deleting a running job, so today a job can be deleted mid-stage while its arq task is still in flight. Again, the legacy pipeline had this.

Those two together are the sharpest finding in the estate: a rewrite dropped two capabilities the thing it replaced already had, and nothing failed as a result, because no artifact represented the old behaviour.

Part.status has six mutation points (upsert_part_error, clear_part_error, finalize_resolve, submit_job_parts, reconcile_submission, mark_submission_stalled). Canon 7's recommendation is derive-where-cheap, else one mutation point. The job level honours that; the part level does not.

Stage logs are latest-run-only by construction (18). A write deletes the stage's prior rows, and the cap is 20 lines truncated to 500 chars. Debugging a job that failed two runs ago is impossible. Canon 18 is open; this is the weakest of the three answers.

No limits or quotas (21). Speedway has checkLimits; this instance has nothing between a caller and the vendor spend.

No machine-to-machine auth (2). Bearer JWT for the SPA only. Confirmed absent by grep across routes/, security.py and config.py; every *_api_key setting is an outbound vendor credential.

Two pipeline generations coexist (35). By accretion rather than design, the same shape as versable-runner's two forks. Canon 35 calls running several versions at once the stated design goal; both instances that do it arrived there by accident and neither can route between them.

No capability discovery (28), no conformance (36), no retention (34).

Worth recording as the opposite of a lapse: this instance has the best outbound rate limiting in the estate (23, a Redis-backed fleet-wide slot limiter per vendor, which canon already names as the shape to copy), the tracing arrangement canon recommends (19, Langfuse behind a no-op default), and the provisioning canon recommends (31, branch to env mapping, one image two services).

7. Unproven#

Inherited from the recon, which flagged them carefully:

• The no-crash-recovery finding rests on absence evidence. Job has no heartbeat or retry column, the only registered stale sweep queries Catalog, and a full-tree grep for heartbeat and stale hits only legacy-pipeline files. What was not verified: whether a worker process death (rather than an in-task exception) actually triggers arq's own timeout-and-retry, or just leaves the in-progress key to expire unwatched. arq's source was not read. That is the next thing to read if this gap matters.
• The no-cancel finding is grep-confirmed, not merely unfound. Cancel machinery exists only against Catalog's status columns.
• API-key auth confirmed absent by grep over the two files that define every auth dependency, though not every router file was opened, so a machine-to-machine path could in principle exist somewhere unread.
• The payload modules' internals (ingest/job_ingest.py, and each *_service_v2.py) were not read in full. Claims like "part-type matching" or "content generation" come from naming and one-line docstrings.

Added this pass:

• Rows 17, 30 and 31 are transcribed from ARCHITECTURE.md and the canon matrix, not from route files I opened. Row 30 in particular lists only /api/build-info; whether a dedicated liveness endpoint exists was not checked, so "health" here may be understated.
• The four Cloud Run services in walmart-496912 were read from live state. The api/worker and prod/non-prod split is inferred from their names and the recon's "one image two services" note, not from deploy config I read.
• Nothing was executed. No job was submitted, no worker was run.