Covers the Python backend of App V5, on Render rather than GCP. Read-only recon of /Users/alcatraz627/Code/Versable/enhancement-product/backend; no files were modified and nothing in it was executed.

Evidence: ../evidence/20260817-enhancement-product-recon.md. Source at /Users/alcatraz627/Code/Versable/enhancement-product/backend. Line references are to that directory unless stated. Read locally 2026-08-17.

1. What it is#

A Mongo-polling pull queue, not a broker, running as WORKER_COUNT (env, default 1) multiprocessing worker processes on Render, each one asyncio loop that claims its own work instead of being pushed to. Four datastores are live at once here, not a legacy-versus-current split: Mongo holds the task and item documents a worker claims and writes its outcome onto (lib/database/__init__.py:24-38), Postgres holds user sessions and the credit ledger (api/auth_db.py:28, lib/jobs/postgres.py:15-34, lib/redis/credit_dispatch.py), Redis holds heartbeats, the fleet-wide retry budget, breaker consensus, the credit-cycle cache and the credit-dispatch queue, and S3 or local disk (switched by STORAGE_BACKEND) holds files. Each store answers a different concern, on purpose.

Two kinds of caller reach it. The frontend sends a bearer-ish X-User-Token header, a base64 blob checked against a Postgres session table (api/auth.py:142-184, api/auth_db.py:28,36). A second class of internal caller uses one shared X-Api-Token, compared with hmac.compare_digest and gated to paths under /internal (api/auth.py:188-206). A separate TypeScript credit-worker consumes a Redis queue this backend writes to; it is downstream of this recon, not examined here.

2. Shape of the runner#

Every worker loop iteration checks two Mongo-native queues before doing any pipeline work: first a cron-like scheduler queue of named, registered tasks in its own scheduler database (lib/scheduler/scheduler.py:53-296), and only if nothing there is due, one random job's next pending item task from the tasks collection, claimed by an atomic find_one_and_update (lib/tasks/claimer.py:22-162,80-81). There is no separate outbox and no broker; worker/worker.md:3-10 states plainly that the earlier Kombu/RabbitMQ design is dead. Once claimed, TaskRunner (lib/tasks/task_runner.py:46-230) loads the task, run, job, pipeline and item in five synchronous Mongo reads run through asyncio.to_thread so the heartbeat keeps ticking (worker/workerv2.py:78-82), validates that the job is not deleted, has not exceeded its retry limit, and the team's credit cycle is active, then calls the payload exactly once and writes the outcome back onto the same task document.

 caller ──POST /jobs──▶ api/jobs.py ──insert per-item──▶ Mongo `tasks`
                                        Task docs
        every worker loop iteration polls in this order:
  ┌── 1. scheduler.scheduled_tasks (Mongo, separate db) ──────────┐
  │      Scheduler.pop_task(), named + registered tasks only       │
  │                                                                 │
  └── 2. tasks (Mongo), only if nothing scheduled is due ◀─────────┘
         AsyncTaskClaimer.claim_random_task()
         random job first, then find_one_and_update flips
         pending → running, stamps lockedAt / lockedBy
                         │
                         ▼
         worker/workerv2.py   (one of WORKER_COUNT processes,
                                one asyncio loop each)
                         │
                         ▼
         lib/tasks/task_runner.py   TaskRunner
         validate() → process()
                         │
             process_pipeline(pipeline, args, item.data, run_id)
                         │
                         ▼
                lib/pipeline/*   (payload, one call in)
                         │
         outcome written back onto the same Mongo task doc:
         result / causes[] / errors[]
 heartbeat:  worker ──5s write──▶ Redis (15s TTL)
             heartbeat_reaper (1-min cron) releases orphaned locks;
             Mongo active_workers is the 30-min backstop + FE roster
 credits:    deduct_credit step ──LPUSH──▶ Redis queue
             ──▶ separate TS credit-worker ──▶ Postgres
             UNIQUE(team_id, cycle_id, credit_key, part_number)
             + ON CONFLICT DO NOTHING

3. Concern by concern#

Rows are canon/00-overview.md. "Vendor" marks mechanisms that only work on one provider or product. A blank mechanism cell means the recon did not examine that concern, not that it is absent; guessing at absence is worse than leaving it open.

4. Runner vs payload#

Mostly clean, with real leaks named rather than smoothed over.

All of worker/*, lib/tasks/*, lib/redis/*, lib/breakers/*, lib/database/*, lib/config/*, cron/*, and the api/tasks.py, api/workers.py, api/redis.py routes are runner. The payload is lib/pipeline/*, invoked through exactly one call: process_pipeline(pipeline, pipeline_args, item.data, run_id) (lib/tasks/task_runner.py:319-320). That single-call boundary is the strongest evidence the seam is intentional; the runner does not know what a pipeline step is.

Four leaks, all acknowledged in the code rather than hidden:

lib/tasks/task_runner.py:145 imports get_active_cycle_for_team from api/auth_db, the runner reaching into api/, normally the payload/HTTP layer, for a Postgres read. The inline comment justifies it as avoiding an "api to lib to api circular" at module load time, a real dependency-direction inversion, deliberate rather than accidental.

Payload code raises runner-typed exceptions to steer runner-level routing: @resilient, CircuitBreakerOpen, RetryBudgetExhausted are runner concerns, but they are raised from inside pipeline methods and caught by task_runner.process()'s inner loop (task_runner.py:341-350). This is a documented contract (lib/tasks/errors.py:1-25), not an accident, but it means payload authors must import runner types.

worker/scheduler_tasks.py:32-35 registers a scheduled task that calls PipelineManager.run_pipeline(**args) directly, a second entrypoint into payload code that runs through the cron-like scheduler queue instead of the item-level claim, lock, retry and heartbeat machinery. It does not get the claimer's lock semantics, the retry budget, or heartbeat-reaper coverage.

deduct_credit is shaped as an ordinary pipeline step, payload-shaped, but its actual effect (dispatching to a Redis queue for a separate credit-worker) is runner- and billing-adjacent, a payload-shaped method carrying a runner-scoped side effect.

5. Deliberate decisions#

Mongo-poll over broker, and the migration is stated as finished, not in-progress. worker/worker.md:3-10 flags the earlier Kombu/RabbitMQ design as dead and warns readers off stale docs or PRs that still mention it. This instance never routes a task through a message broker.

Two-tier heartbeat is a design choice, not an accident of two systems existing. Redis is the source of truth for second-by-second liveness; Mongo carries the historical roster (lib/redis/worker_heartbeat.py:8-9). The 30-minute Mongo lock-timeout is an explicit backstop for when Redis itself is the thing that is down.

Credit idempotency lives in a database constraint, not app-level dedup. .claude/notes/credit-flow.md:52-54 states plainly that ON CONFLICT DO NOTHING is the design, not a workaround, and asks future readers not to re-litigate it.

Quota checks are deliberately lazy. The same note (credit-flow.md:44-50) documents a pre-job check plus an async idempotent charge, with no synchronous budget check at pipeline-apply or task-dispatch time. That is a named speed/complexity/correctness trade, with the residual risk window written down rather than left implicit.

Breaker-outermost, retry-innermost ordering is fixed by a decorator, not left to callers. lib/breakers/decorator.py:1-11 locks the order per resilience4j convention specifically so callers cannot invert it.

Claim-time fairness picks a job before it picks a task. distinct on job_id first, then find_one, so no single large job can starve every other job's items out of a worker's attention (claimer.py:80-81).

deferred is a real outcome, distinct from retry and from failure. CircuitBreakerOpen and RetryBudgetExhausted route to a deferred state that costs the item no attempt at all (task_runner.py:341-350), a third category most of the estate does not have.

Queue namespacing by git branch gives PR previews zero-manual-config isolation on one shared Mongo instance (lib/config/__init__.py:88-93).

One cron job deliberately carries two unrelated jobs, notification and the heartbeat reaper, because Render cron creation is heavy and the cadence happens to match (cron/notify_jobs.py:99-101).

6. Lapses#

Written against canon, not as bug reports.

Caller identity (2) is the same open weak point it is everywhere else in the estate, in a slightly different shape. One shared X-Api-Token covers every internal/machine caller (api/auth.py:188-206), compared with constant-time equality but with no per-caller identity, no revocation short of rotating the single value for everyone, and no JWT or IdP path at all for machine-to-machine calls.

Roles (4) do not exist below the team boundary. Ownership is a flat team_id plus user_id, and the one privileged role is a single hardcoded admin team whose members bypass every scoping filter (api/auth.py:17-18,62-65,80-85). Canon leaves roles as app-owned and recommended, but this instance has no role concept to hand a module scopes from.

The runner/payload seam (1) has two real leaks alongside its clean single-call boundary. The dependency-direction inversion at task_runner.py:145 and the second, unguarded entrypoint into payload code at worker/scheduler_tasks.py:32-35 both mean scheduled pipeline runs can bypass the claimer's lock semantics, the fleet-wide retry budget, and heartbeat-reaper coverage, the exact protections the item-task path was built to guarantee.

No per-item log stream a caller can read (18), same gap as versable-runner. PrintLogger writes to stdout only; usage-style sidecars comparable to versable-runner's were not found for this instance. Canon calls 18 open, and this instance contributes no answer either.

No versioning at all (29). No contract version, no payload schema version, no module version a caller can read. The only build-identity signal found is a commit-based Slack dedup key for preflight, which is internal tooling, not a caller-facing surface. speedway and walmart-mvp both report a build commit to callers; this instance was not found to.

No capability discovery (28). Nothing comparable to versable-runner's self-documenting GET /usage was located in the files read.

No per-item paginated results endpoint (17) was located, only a job-level live aggregate. Canon settles pagination and per-item fetch as mandatory on every list; whether this instance actually lacks it, or the recon simply did not find it, was not resolved.

7. Unproven#

• The circuit-breaker fleet-consensus mechanism (lib/breakers/consensus.py) and its design doc were referenced by several files but not opened. The quorum/threshold behavior is inferred from local.py's docstring and two config vars only, not verified against consensus.py itself.
• lib/breakers/notify.py and lib/breakers/status.py were not opened. Their existence and role are taken from lib/breakers/__init__.py's module docstring.
• lib/pipeline/manager/__init__.py and lib/pipeline/processor/__init__.py were located and sized but not read line by line. The single-call boundary claim in section 4 is confirmed only from its call sites in task_runner.py, not from the function's own implementation.
• The frontend BTS doc's path moved from boring-technical-stuff/ to tech/ at some point; the current file (frontend/docs/tech/backend/workers/_index.md, updated 2026-06-17) was found and read only after the originally cited path came up missing. worker/worker.md's own pointer to the old path is stale.
• lib/jobs/__init__.py is roughly 73KB (JobsManager), grepped for specific call sites only, not read in full. Job creation, deletion, and pause/resume internals are known only from their api/jobs.py call signatures.
• lib/jobs/formats/cache.py (job_output_cache) was referenced via its invalidation call sites but not opened; its caching mechanism is inferred only from env var names.
• No RBAC table beyond team_id/user_id plus one hardcoded admin team was found in the files searched. Roughly 35 lib/ submodules (lib/agents/, lib/matchers/, lib/providers/, lib/scraper/, lib/importer/, lib/exporter/, among others) were not opened at all; this is an absence in what was searched, not a certainty that nothing exists anywhere in the payload-side tree.
• Rows 15, 16, 33 vendor marks (S3 vs local disk). STORAGE_BACKEND switching the files collection name was read; whether output delivery genuinely shares that same switch, rather than a separate path, was not confirmed.
• Nothing here was executed. Every claim is read from source. No job was submitted, no worker was started, no pipeline was run.

8. Retrospective: the assumptions this system was built on#

Owner-stated on 2026-08-18, recorded here so the instance carries its own reasoning and not only its mechanisms (../evidence/20260817-owner-answers.md §11).

• As few services as possible. Render plus Vercel, and every new capability added to an existing service. This bought one runner, one auth, one credits system, one observability stack, one deploy, and it meant no capability was ever reimplemented. It cost architectural headroom: "what we had was what all we had", one capability's resource profile became everyone's ceiling, and a new shape needed an architectural change.
• Composability in-system. The pipeline could compose steps, but only its own steps. Integrating the external extractor, with its own job lifecycle, fought the model instead of plugging in. Under the contract, an outside system is a module with the same surface as an inside one.
• The pipeline shape was the only shape. When M output rows were needed for N input rows, "the pipeline straight up did not support" it, and the job output layer was built. That layer then carried exports and on-the-fly, non-pipeline modifications, and let pipeline rows re-run or fail at their own pace while outputs read whatever was settled. The owner's summary, "mutable heavy agentic worker run plus declarative output transform run", is now ../canon/15-outputs-and-transforms.md and ../guides/01-outputs-as-a-transform.md.
• A particular split of responsibility and source of truth across UI, BFF, backend API, and the hand-rolled worker scheduler: Postgres for user-facing data, Mongo for system I/O and processing, with job templates the owner names as misplaced and configuration "always hanging around in a weird place". The data-model split across this and the sibling apps is being examined for ../guides/02-data-ownership.md (../evidence/20260818-data-model-split/).

The general lesson is in ../guides/00-service-granularity.md: build the seams before you need them. Every seam in the contract is one this system needed after the fact.