Figure 6 · E004

Context tokens per correct answer

This divides a run's context tokens (Figure 1) by the number of questions it answered correctly, so an arm that reads less but also answers less is not rewarded.

protocols
battery
model
columns

Pooled models shows one column per arm, its models averaged with equal weight — the readable overview. Per model shows one column per model for every arm, for like-for-like reads at a fixed model. This choice applies to every figure on the site and is remembered.

Context tokens per correct answerlower is betterZeroShot + MultiTurn · both batteries
0200k400k600kCC Bare 454kCodex Bare 245k★448kHaiku 4.5★156kSonnet 5★126kOpus 4.6★134kOpus 4.8★140kOpus 5★63kFable 5★232kLuna★148kTerra131kSolTrueArchitectTrueArchitect454k6 modelsCC Barebare harness3rd of 7245k3 modelsCodex Barebare harness1st of 7712kHaiku 4.5439kSonnet 5366kOpus 5331kLuna167kTerra★113kSol★312kGrok 4.6★432kComp 2.5Cursor Barebare harness467k6 modelsGraphifyindexing tool6th of 7465k6 modelsGitNexusindexing tool5th of 7359k6 modelsCodeGraphindexing tool2nd of 7457k6 modelsCodebaseMemoryindexing tool4th of 7502k6 modelsSerenaindexing tool7th of 7

Raw token counts are never pooled across vendors (tokenizers differ), so arms whose selected models span vendors are shown one column per model; ★ marks the best value at each model.

Data behind this figure — ZeroShot + MultiTurn · both batteries
armrolecolumncontext tokens per correct answerrunscellsminmaxsdrankruns in the repository
TrueArchitectTrueArchitectHaiku 4.5 ★448k39484k1.35M378k—Haiku 4.5
Sonnet 5 ★156k40451k399k109kSonnet 5
Opus 4.6 ★126k40427k434k115kOpus 4.6
Opus 4.8 ★134k40429k425k122kOpus 4.8
Opus 5 ★140k40446k311k78kOpus 5
Fable 5 ★63k11226k106k34kFable 5
GPT 5.6 Luna ★232k40429k890k254kGPT 5.6 Luna
GPT 5.6 Terra ★148k40422k509k154kGPT 5.6 Terra
GPT 5.6 Sol131k40422k455k129kGPT 5.6 Sol
CC Barebare harness6 models454k20122112k2.17M382k3rd of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
Codex Barebare harness3 models245k1201272k901k165k1st of 7GPT 5.6 Luna · GPT 5.6 Terra · GPT 5.6 Sol
Cursor Barebare harnessHaiku 4.5712k40480k1.49M396k—Haiku 4.5
Sonnet 5439k404172k988k243kSonnet 5
Opus 5366k404219k865k158kOpus 5
GPT 5.6 Luna331k404157k823k161kGPT 5.6 Luna
GPT 5.6 Terra167k40470k438k94kGPT 5.6 Terra
GPT 5.6 Sol ★113k40466k369k53kGPT 5.6 Sol
Grok 4.6 ★312k404143k594k140kGrok 4.6
Composer 2.5 ★432k404179k843k205kComposer 2.5
Graphifyindexing tool on Claude Code6 models467k20122119k2.04M380k6th of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
GitNexusindexing tool on Claude Code6 models465k20122163k1.95M344k5th of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
CodeGraphindexing tool on Claude Code6 models359k2032297k1.97M288k2nd of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
CodebaseMemoryindexing tool on Claude Code6 models457k20122136k1.76M330k4th of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
Serenaindexing tool on Claude Code6 models502k20723138k2.09M417k7th of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5

Figure 6. This divides a run's context tokens (Figure 1) by the number of questions it answered correctly, so an arm that reads less but also answers less is not rewarded. Each column is the mean over valid, scored runs at one model (or pooled over a single-vendor roster in the pooled-models view), lower is better, and TrueArchitect and the bare harnesses are the dashed references. The ordering of Figure 1 survives the normalisation: the token saving is a saving per correct answer, not a saving bought with wrong answers.

How this figure is computed

Definition. For a scored run with score(r) > 0, ctx_per_correct(r) = context(r) / score(r). Runs with no correct answer have no defined value and are excluded from the mean; their count is in the table.

Aggregation. As in Figure 1: cell means averaged with equal weight; raw token counts are never pooled across vendors.

Reading it. Lower is better. Cost in currency is deliberately not shown here: the package publishes cost only where a vendor reported it, and TrueArchitect's runs carry no vendor-reported figure. Tokens are the quantity a reader can recompute from the run records.

Provenance

Computed at build time from epochs/E004/summary/runs.json of the proof package (export tool tabench export public 0.1.25, scorer era tabench-1.1). Each row of the table above links to the run directories in the repository, where run.json, the answers, the verdicts and the transcript of every run can be read.