Covers services-api, runner-service, and runner-service-dev, which are three Cloud Run services in GCP project versable-runner running the same runner over different method registries.

Evidence: ../evidence/20260817-runner-architecture.md, 20260817-runner-blueprint-approach.md, 20260817-runner-four-file-diff.md. Source is now the git repo at /Users/alcatraz627/Code/Versable/services-api, pinned at f46d894. Line references are to that repo unless stated. /Users/alcatraz627/Code/Versable/gcp/src/ is kept as the deployed baseline 2026-08-17 = 23a0240, recovered from the Cloud Run source-deploy bucket; it is not deleted and not refreshed. Three commits landed on 2026-08-18 after that baseline (5c0ec82 S3 to GCS, fc26956 merge, f46d894 vsk_ keys), so a claim sourced from the baseline alone is stale by that much. Live GCP state read 2026-08-17. Drift analysis: ../evidence/20260818-services-api-git-source.md.

1. What it is#

A pure module. No users, no UI, no human-facing surface at all. A caller posts a list of items and a method name over HTTP, every item becomes its own Cloud Tasks task pushed back to the same service, and per-item results land in GCS. Python on Cloud Run, us-central1, 1 vCPU and 2Gi, maxScale 5. GCS is the only datastore: no Mongo, no Redis, no SQL anywhere in the request path.

Callers today are internal and unnamed. The service cannot tell them apart (concern 2), so "who calls it" has no answer beyond "whoever holds the password".

Three deployments share one queue-and-bucket pattern:

2. Shape of the runner#

Submit is O(1) in item count. POST /jobs writes two blobs and enqueues exactly one fanout task, so a 5,000 item job returns its job_id immediately. The expansion into per-item tasks happens inside the queue, on the fanout task, and the job reports enqueuing until that finishes.

 caller ──POST /jobs──▶ services-api ──1 fanout task──▶ Cloud Tasks
                             ▲                              │
                             │                              │ POST /internal/
                             │                              │  tasks/fanout
                             └──────────────────────────────┘
                                        (self-push)
                             ▲                              │
                             │      N per-item tasks        │
                             └──────────────────────────────┘
                                POST /internal/tasks/process-item
                                          │
                                          ▼
                          GCS  results/{idx}.json   on success
                               errors/{idx}.json    on terminal failure
                               usage/{idx}.json     always
 state = count(manifest items) vs count(results) + count(errors)

The service pushes tasks to itself. There is no separate worker deployment.

3. Concern by concern#

Rows are canon/00-overview.md. "Vendor" marks mechanisms that only work on one provider.

Vendor coupling concentrates in exactly three places: Cloud Tasks for dispatch and concurrency, GCS for storage and idempotency, and Secret Manager. The runner's own logic is portable; app/dispatch.py already proves it by shipping a second, queueless backend.

4. Runner vs payload#

The seam is app/ against lib/, and it is clean.

All 12 files in app/ are runner. ../evidence/20260817-runner-four-file-diff.md settles this: the four files that differ between services-api and runner-service differ only by version skew, with services-api a strict superset. Routes, request shapes, job states, storage layout, auth and dispatch are identical across the fork. Nothing in app/ is per-service.

The domain lives entirely in lib/ plus the method registry, and for services-api also data/ (SQLite PCdb, vectors) and assets/ (a 122MB vectors.npy, a 45MB records.jsonl, a 17MB sqlite baked into the image).

No leaks found in the runner direction. app/ does not import domain concepts. The seam held across an independent fork with nobody enforcing it, which is stronger evidence than a design review.

One leak in the other direction: lib/ carries Langfuse tracing, so the payload owns an observability decision that concern 19 places in the runner.

Caveat on the strength of this evidence, from the same diff doc: the two forks run the same runner over different method registries, so the fork demonstrates the seam holds but says almost nothing about what varies between genuinely different services. That has to come from speedway and walmart-mvp.

5. Deliberate decisions#

State is derived, never stored (app/jobs.py:138-148). Status is a count of blobs against the manifest. No status column to go stale, no state machine to corrupt, no write contention on a job record, and a crashed process leaves no lock. This is the load-bearing decision; most of the others follow from it.

Submit is O(1) in item count. One fanout task expands into N. The cost is a genuinely reachable enqueuing state that every caller must handle.

The handler owns the retry budget, and the queue is only a backstop (app/jobs.py:337-338). On exhausting RUNNER_MAX_ATTEMPTS the handler writes a terminal error blob and returns success, so the queue stops redelivering. The queue's own limit of 5 only applies when the handler dies without answering. RUNNER_ITEM_TIMEOUT_S (1700) sits under the 1800s dispatch deadline for the same reason.

Idempotency is structural rather than defensive (app/store.py:123). Deterministic task names plus if_generation_match=0 mean a duplicate delivery cannot clobber a finished result. That is what makes the fanout step safely redeliverable without a transaction.

Zero-env import (lib/config/__init__.py:103-122). Every config var has a default, so the app imports and runs with no environment set. This is what makes RUNNER_DISPATCH=local a real development mode rather than a claim.

A machine-readable capability surface (app/usage.py:177). GET /usage generates plain-text documentation from the live method registry, aimed at LLM agents. Nothing else in the estate does this, and concern 28 is otherwise open.

Divergences from contracts/caller-keys.md, all deliberate#

f46d894 (2026-08-18) added per-user vsk_<key_id>_<secret> keys, minted, listed and revoked in prompt-mgmt and validated by this module over HTTP. That is a third identity shape: not jwks, and not the module-issued smk_ keys of option C. guides/03 carries it as contender F.

Every row below is pinned by a test, so each is a decision someone made and wrote down, not an oversight. The witnesses live in tests/test_auth.py and tests/test_apikeys.py.

Two of its choices are better than the contract had, and caller-keys.md § The verifier has adopted both. A key-service outage is never cached as a rejection, so prompt-mgmt going down cannot lock a valid key out for the TTL (apikeys/client.py:38, witness test_outage_rejects_but_is_not_cached). And a credential that is not vsk_-shaped never reaches the network, keeping the password path off the wire (app/auth.py:44, witness test_non_key_shaped_credentials_never_call_the_service).

The RUNNER_API_PASSWORD row is the one that stays a lapse. The others are a different identity model the contract now names and can rule on. That row is the shared password the contract refuses outright, still live beside the keys, and no ruling makes it conforming.

Storage moved to GCS behind a factory#

5c0ec82 (2026-08-18) migrated object storage from AWS S3 to GCS behind a central use_cloud_storage factory (lib/providers/storage/__init__.py:56, with lib/providers/storage/gcs.py new). Nothing in the contract breaks: contracts/manifest.md already allows runtime.storage of gcs, s3 or local. Rows 15, 16 and 22 above describe GCS and remain correct; what changed is that the choice is now a seam rather than a hard-wired provider.

6. Lapses#

Written against canon, not as bug reports.

Caller identity (2) is the weakest point in the estate, and f46d894 narrowed it without closing it. Per-user vsk_ keys now give real per-caller identity, revocation without rotating a shared value, and a user_id for attribution (app/auth.py:44-48). What remains: RUNNER_API_PASSWORD is still accepted beside them (app/auth.py:42), so one shared password still opens every door, and the contract refuses that outright. Ingress is still public with app-level auth rather than IAM. The verdict carries no tenant or scope, so authorization is all-or-nothing once a key validates. The meta object records {service, organization, user} and nothing verifies any of it.

That services-api grew X-Meta-<key> headers as a second way to pass tags (per the four-file diff) is worth reading as a symptom: the runner is being asked for caller identity and answering with self-declared tags.

Tenancy (3) does not exist. Canon settles tenant as mandatory on every job. Here it is an unverified string. Both other instances enforce it; this one is the outlier.

No per-item log stream (18). usage/{idx}.json records duration, attempts and tokens, but a caller debugging one failed item out of 5,000 has the error type and message and nothing else. Canon calls 18 open; this instance contributes no answer.

No versioning at all (29). No contract version, no module version, no payload schema version, and no build identity endpoint. Both other instances at least report a commit SHA. A caller cannot tell which version answered it. Confirmed live: GET /build-info and GET /manifest both return 404.

/health returns the wrong shape. Live response is {"ok":true}; contracts/module-surface.md specifies {status:"ok"}. A one-line divergence, and worth fixing before other modules copy the running service rather than the contract. patterns/03 carries it in the parity ledger.

Dev shares prod's service account. runner-service-dev runs as runner-service@, so the two share blast radius and neither can be audited separately. Canon 31 asks for per-environment identity.

Self-push conflates API and worker capacity. The service enqueues to itself and handles its own tasks, so one maxScale 5 covers both roles. An expensive item pipeline can starve the submit path, and the remedy is a topology change rather than a config change. Fine at current size; it is a ceiling, not a bug.

services-api and runner-service share one queue, so they share one semaphore. Both are configured onto pipeline-runner, and maxConcurrentDispatches is the only throttle in the system, so the two services compete for one global concurrency budget with no fairness between them. Neither can be throttled without throttling the other, and a burst in one starves the other silently. Only runner-service-dev has its own queue.

No limits or quotas (21), no outbound rate limiting (23), no retention (34). A single caller can saturate the queue, and nothing ages data out of the buckets.

7. Unproven#

• That services-api and runner-service are parallel pipelines rather than two tiers of one. Circumstantial: RUNNER_SERVICE_URL on services-api points at itself, they write to different buckets, and one owns lib/ while the other syncs it. Not traced through app/dispatch.py end to end.
• That there is no CI. gcloud builds triggers list is empty and deploys run from a workstation via deploy.sh, but not every surface was checked.
• Row 19 (tracing). Langfuse was located by grep in lib/providers/gemini/ and lib/pipeline/methods/; the claim that keys default empty is read from config convention, not from an executed no-op path.
• Row 25 (secrets). Read from live Cloud Run state, which shows 8 secrets injected by reference with no literals. Secret values were never read, by design, so nothing is known about their contents or rotation.
• Rows 9 and 14 vendor marks. maxConcurrentDispatches and if_generation_match are Cloud Tasks and GCS features respectively. Whether equivalents exist on other providers was not researched.
• Nothing here was executed. Every claim is read from source or from live GCP state. No job was submitted, no code path was run.