Epoch 004 — line-4 · public proof package

Does TrueArchitect's codebase index make an AI coding agent more accurate, more consistent and cheaper than text search?

This site reports one benchmark epoch in which the TrueArchitect Ask agent — the model plus TrueArchitect's index of the codebase, built by its own proprietary indexing system and designed for LLM consumption — answered 50 questions about usememos/memos under three exam protocols, beside bare Claude Code, bare Codex, bare Cursor Agent, and five codebase-indexing tools installed on Claude Code, at the same models and from the same clean checkout. Every figure is computed from the run records in the public repository, and every number links to the runs behind it.

The proof package on GitHub — TrueArchitect-ai/TrueArchitect-Benchmarks2,692 run directories: every question, every answer, every verdict with the judge's reason, every transcript, and the export that produced them. Every number on this site links into it.

2,692

runs published
2,480 valid and scored · 212 did not finish

arms
1 TrueArchitect · 8 comparison

models
Anthropic, OpenAI, xAI, Cursor

61,770

question verdicts
one scorer era: tabench-1.1

Key results — ZeroShot and MultiTurn protocols, both batteries (E004)

Context. At every model both arms ran, bare Claude Code read 1.9× to 2.9× the context tokens TrueArchitect read to answer the same battery (Haiku 4.5: 8.75M vs 16.53M; Sonnet 5: 3.37M vs 9.69M; Opus 4.6: 2.63M vs 7.03M; Opus 4.8: 2.86M vs 5.51M; Opus 5: 3.24M vs 6.37M; Fable 5: 1.90M vs 4.66M). — Figure 1
Accuracy. TrueArchitect's pooled accuracy is 93.3% against 89.4% for bare Claude Code and 90.6% for the best indexing tool (GitNexus). — Figure 2
Reliability. 77% of TrueArchitect's runs scored at least 90%, against 67% for bare Claude Code. — Figure 3

Pooled figures average protocol × battery × model cells with equal weight; see each figure's How this figure is computed. Change protocols, battery and model on every figure page. HumanExam (the whole battery in one prompt) is off by default because it is the stress probe, not the usage shape; switch it on to see it.

Headline figure

Context tokens per runlower is betterZeroShot + MultiTurn · both batteries

Raw token counts are never pooled across vendors (tokenizers differ), so arms whose selected models span vendors are shown one column per model; ★ marks the best value at each model.

Figure 1. Context tokens are the tokens the model read to answer a whole run: the uncached input plus every cache read, summed over every API call in the run, all threads included. Each column is the mean over the valid, scored runs of one arm at one model, every arm split by model because token counts are model facts; the dashed lines mark TrueArchitect's and each bare harness's pooled value wherever pooling is licensed. At every model the TrueArchitect column sits well below the same model running bare, and the indexing tools cluster at or above bare Claude Code: the index replaces reading. Controls, data table and method →

The figures

Figure 1Context tokens per runAt every model the TrueArchitect column sits well below the same model running bare, and the indexing tools cluster at or above bare Claude Code: the index replaces reading.Figure 2Accuracy per runTrueArchitect's distribution sits highest and tightest; the lower context use in Figure 1 is not bought with accuracy.Figure 3Reliability: runs scoring at least 90 percentThe gap between arms is far wider here than in mean accuracy: the index converts a good average into a dependable result.Figure 4Effect size against bare Claude CodeBy convention d of 0.2 is small, 0.5 medium and 0.8 large; TrueArchitect is the only arm above the large threshold.Figure 5Outcome composition per runThe dark bands are the honesty layer: the failures are on the page, not in a footnote.Figure 6Context tokens per correct answerThe ordering of Figure 1 survives the normalisation: the token saving is a saving per correct answer, not a saving bought with wrong answers.Figure 7Tool calls per runFewer calls with higher accuracy is the signature of a good index: the model asks its question once instead of searching for the answer many times.Figure 8Wall time per runTiming is environment-coupled and reported as indicative magnitude only; see the disclosures below before comparing across the host boundary.Figure 9Accuracy by modelThe comparison to make is diagonal: TrueArchitect at a fast or everyday model against a bare harness or an indexing tool at a frontier model.Figure 10Pass rate by question category and difficultyThe categories where an index should matter most, structural questions such as callers, impact and cross-stack tracing, are where the separation is widest.Figure 11Cost per correct answerThe context saving of Figure 1 becomes a price saving at every model, and the ordering survives the normalisation by correct answers.Figure 12Cost per runA tight, low cloud is an arm whose price is predictable; a tall cloud is one whose price depends on the run.Figure 13Tool result tokens per runThis is the mechanism behind Figure 1 seen from the tool side: the index returns a small, precise answer where a search returns pages to read.

How to read this site

An arm is one configuration of tooling; a model is the LLM behind it; a run is one repetition of one arm × model × exam protocol × battery. Runs are independent sessions. Means are taken over valid, scored runs only; a run that did not finish is published unscored and never enters a mean.

Every figure page has three parts: the figure with its controls, the formal definition of what was computed, and the data table with a link per row into the repository. The Method page is the repository's README rendered; the Limitations page collects every disclosure in one place.