Verify-Loop CI for Agent-Written Code

Once agents author most code and PR review speeds up, CI is the constraint. Cite Anthropic’s public numbers only: 25x CI jobs in 6 months, Claude ~80% of shipped code, 8x ship rate vs 2021–2025, tests 10x. Do not invent Institute throughput or revenue figures. Smaller, more granular agent PRs and overnight/weekend agent activity raise the CI floor; bursts still follow human approval.

Do not run every test on every change. Anthropic’s service is a deterministic test-impact / selection layer: a listener records results from every CI run; a selector reads history and package relevance to decide which tests run on which opened PRs. Agents need a specific valid set to self-verify; humans can ignore irrelevant failures, agents cannot. Vendor test-selection products exist in this category — build or buy, same split.

A single-process listener/selector cannot shard: one writer holding per-test history blocks horizontal scale. Anthropic’s path: patches (bigger machine, then per-package shards, then daily restarts) bought less time each round. The durable design offloads history to a store: any listener worker appends a job-result to a journal and stays stateless; a small consumer rolls the journal into per-test history; the selector looks up history. Do not keep the running selector state inside the process.

Listener lag is the failure mode: stale selector data runs the wrong tests (already-flaky or already-failing suites) and misses new/fixed tests. Anthropic’s operator metric: the same number of CI jobs coming in should equal the same going out. Instrument queued unprocessed job-result events. Use those signals as the agent’s eyes so it can hill-climb incrementally. A lagging listener is a verify-loop bug, not a reason to skip CI.

Anthropic’s advice, from the same public post: whether you build or buy, assume the architecture will be at a 25x load within two quarters. Account for 10–20x perceived scale in v0 if budget allows. Quick patches buy a fraction of the time they did a year ago; rewriting the service also takes a fraction of the time now that authoring is not the bottleneck. The insight is not “buy a bigger box” — it is plan the exponential before the singleton pages.

Dario Amodei’s pacing essay (https://darioamodei.com/post/we-must-pace-the-frontier) commits Anthropic to embedded third-party evaluators (example: METR) with employee-like access to verify safety practices and report incidents. Use that here only as optional instrumentation context: expose jobs-in = jobs-out, selector lag, and selection coverage so an embedded reviewer can see the verify loop. This playbook does not restate the essay’s coordination steps. METR Inspect Hawk remains the catalog eval harness — separate from test selection.