What this benchmark does not show, and what it shows with caveats

Everything below is also stated where it applies; this page collects it so nothing has to be found.

Scope

One repository under test, at one commit, with two question batteries authored by the benchmark's owner. The questions are about code structure and behaviour (endpoints, schema, types, call paths, impact, cross-stack tracing, absence proofs, false premises); they are not a coding task benchmark and say nothing about editing or generating code.

One TrueArchitect instrument freeze per epoch. The comparison group is measured as shipped, prompt assets and all; Claude Code cannot be forced to use an installed tool, so what is measured for the indexing tools is organic adoption. That is the measurand, not a limitation of the measurement.

Execution asymmetries

The TrueArchitect arm executed on the host through its gateway; the comparison group executed in containers on the same machine. Accuracy, tokens, turns and tool counts are location-independent; wall-clock time is not, and Figure 8 is published for completeness rather than as a claim. Runs executed in parallel lanes shared the host; per-lane contention is not modelled.

Cursor Agent's model traffic transits Cursor's backend, and Codex runs inside its own harness loop; both are measured as their vendors ship them. Their token figures are the harness's own per-call usage with the derivation named in each run record.

Tokens and cost

Raw token counts are never pooled across vendors, because tokenizers differ; figures split by model whenever a selection spans vendors. Cost is the vendor-reported figure where the harness reported one (Claude Code's own result envelope) and otherwise an estimate from the published rate tables (epochs/…/summary/rate-tables.json) over the run's four token columns — list prices, so an arm on a subscription plan may pay less in practice; every run row names its cost_basis, and a model with no rate row has no cost and never enters a cost mean. Codex reports no cache-write split, so its estimate charges written tokens at the input rate, a disclosed floor.

Scoring

Tier T1 is a mechanical token-subset match that can only pass an answer. Tier T2 is an LLM judge under a fixed prompt with per-question exemplars, one candidate per call, whose reason is published beside every verdict; a verdict whose reason contradicts it is retried once and, if still contradictory, marked adjudicate and counted as a fail. The judge is an LLM and can be wrong; the per-question answers and verdicts are published so that any reader can re-grade. Every verdict in the epoch sits under one scorer era.

Did-not-finish runs

A run that did not complete every question is published with validity.ok = false and no score. It never enters a mean. The counts per arm in this epoch:

A partial exam is most often a single question that never completed in a per-question protocol; the run's validity.detail names it. By model and protocol:

Model-specific caveats

Fable 5 refused many questions. At Fable 5 the model declined to answer a large share of the per-question prompts — plain questions about a public codebase's structure, nothing touching security — so those runs did not finish and carry no score: 79 of 119 ZeroShot runs at Fable 5 are did-not-finish across the arms that ran it. Fable 5 is therefore a thin roster in this epoch (23 TrueArchitect runs in total) and its columns should be read with their n; it is excluded from nothing, but it decides nothing either.

Haiku 4.5 and the whole-battery protocol. HumanExam presents the entire battery in one prompt, and at Haiku 4.5's 200K-token context window the accumulated tool results of a full battery can exceed the window. 5 of 25 HumanExam runs at Haiku 4.5 did not finish for that reason (all 5 on the TrueArchitect arm, whose index results are large). The per-question protocols do not have this exposure and are the ones to read at Haiku 4.5.

Opus 4.8 in the ZeroShot protocol. The comparison group's ZeroShot runs at Opus 4.8 stop after ten valid repetitions per arm with twenty did-not-finish runs each; the run records carry the failing question in validity.detail. Those runs are published unscored like every other did-not-finish run and never enter a mean.

Capture and provenance

Capture gaps are disclosed per run in run.json and are never silently filled. Runs whose units were resumed across hosts or image builds list every vintage. The capture host's name appears in provenance; home directories are normalised; nothing else about the operators' machines is published.

What is withheld

The text of the queries the TrueArchitect agent sends to its codebase index and the raw result envelopes, replaced in transcripts by the product's own display sentence with latency and result size; and the internal code names of TrueArchitect components. Comparison-group transcripts are verbatim. The full policy is redaction/POLICY.md in the repository.

Per-figure disclosures

Figure 1 — Context tokens per run

Exam protocols have different token denominators (a MultiTurn question replays its history; ZeroShot pays the fixed prefix per question). Switch protocols with the control above; the site never pools across them.

A run whose token facts were not captured is excluded from its column, never counted as zero; the table's n is the number of runs behind each column.

Figure 5 — Outcome composition per run

Did-not-finish runs have no score and do not appear here; their count per arm is in the run records and in the Limitations page.

Figure 8 — Wall time per run

The TrueArchitect arm executed on the host through its gateway; the comparison group executed in containers on the same machine. Accuracy, tokens and tool counts are location-independent; wall time is not, and the figure is published for completeness rather than as a claim.

Runs executed in parallel lanes shared the host; per-lane contention is not modelled.

Figure 11 — Cost per correct answer

Vendor-reported and estimated costs sit in the same column; the table beneath the figure carries the basis per row and the rate tables are published beside the run rows.