Figure 13 · E004

Tool result tokens per run

Tool result tokens are the volume of text the arm's tools pushed back into the model's context over a run — file contents, search hits, shell output, or an index query's result — estimated per call from the result size.

protocols
battery
model
columns

Pooled models shows one column per arm, its models averaged with equal weight — the readable overview. Per model shows one column per model for every arm, for like-for-like reads at a fixed model. This choice applies to every figure on the site and is remembered.

Tool result tokens per runlower is betterZeroShot + MultiTurn · both batteries
0500k1.0MCC Bare 125kCodex Bare 240k170k59k292kHaiku 4.580k21k151kSonnet 576k26k132kOpus 4.666k23k101kOpus 4.897k42k162kOpus 548k21k77kFable 5269k119k413kLuna186k77k269kTerra178k97k276kSolTrueArchitectTrueArchitect125k3k536kCC Barebare harness4th of 7240k35k496kCodex Barebare harness7th of 7142k42k617kHaiku 4.574k19k156kSonnet 596k25k370kOpus 5301k44k883kLuna453k50k1.29MTerra718k52k1.22MSol210k66k585kGrok 4.6248k89k600kComp 2.5Cursor Barebare harness130k10k517kGraphifyindexing tool6th of 789k10k527kGitNexusindexing tool2nd of 7128k7k414kCodeGraphindexing tool5th of 782k10k345kCodebaseMemoryindexing tool1st of 7109k9k445kSerenaindexing tool3rd of 7

Raw token counts are never pooled across vendors (tokenizers differ), so arms whose selected models span vendors are shown one column per model; ★ marks the best value at each model.

Data behind this figure — ZeroShot + MultiTurn · both batteries
armrolecolumntool result tokens per run (estimated)runscellsminmaxsdrankruns in the repository
TrueArchitectTrueArchitectHaiku 4.5170k39459k292k83k—Haiku 4.5
Sonnet 580k40421k151k43kSonnet 5
Opus 4.6 ★76k40426k132k34kOpus 4.6
Opus 4.8 ★66k40423k101k25kOpus 4.8
Opus 597k40442k162k40kOpus 5
Fable 5 ★48k11221k77k22kFable 5
GPT 5.6 Luna ★269k404119k413k82kGPT 5.6 Luna
GPT 5.6 Terra ★186k40477k269k57kGPT 5.6 Terra
GPT 5.6 Sol ★178k40497k276k48kGPT 5.6 Sol
CC Barebare harness6 models125k201223k536k147k4th of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
Codex Barebare harness3 models240k1201235k496k156k7th of 7GPT 5.6 Luna · GPT 5.6 Terra · GPT 5.6 Sol
Cursor Barebare harnessHaiku 4.5 ★142k40442k617k112k—Haiku 4.5
Sonnet 5 ★74k40419k156k45kSonnet 5
Opus 5 ★96k40425k370k74kOpus 5
GPT 5.6 Luna301k40444k883k222kGPT 5.6 Luna
GPT 5.6 Terra453k40450k1.29M280kGPT 5.6 Terra
GPT 5.6 Sol718k40452k1.22M368kGPT 5.6 Sol
Grok 4.6 ★210k40466k585k132kGrok 4.6
Composer 2.5 ★248k40489k600k146kComposer 2.5
Graphifyindexing tool on Claude Code6 models130k2012210k517k144k6th of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
GitNexusindexing tool on Claude Code6 models89k2012210k527k83k2nd of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
CodeGraphindexing tool on Claude Code6 models128k203227k414k111k5th of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
CodebaseMemoryindexing tool on Claude Code6 models82k2012210k345k83k1st of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
Serenaindexing tool on Claude Code6 models109k207239k445k111k3rd of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5

Figure 13. Tool result tokens are the volume of text the arm's tools pushed back into the model's context over a run — file contents, search hits, shell output, or an index query's result — estimated per call from the result size. Every dot is one run, split per model for every arm; the solid tick is the mean and the thin line the range. This is the mechanism behind Figure 1 seen from the tool side: the index returns a small, precise answer where a search returns pages to read.

How this figure is computed

Definition. tool_result_tokens(r) = Σ over the run's tool calls of the result's token estimate, taken from the capture where the harness recorded one and otherwise result bytes ÷ 4. For TrueArchitect index queries the estimate is the size of the result the model received.

Aggregation. None for the dots; the mean tick is the equal-weight mean of battery × model × protocol cells. Raw token counts are never pooled across vendors.

Reading it. Lower is better. Read it beside Figure 7 (tool calls): fewer calls returning less text is the signature of a good index.

Provenance

Computed at build time from epochs/E004/summary/runs.json of the proof package (export tool tabench export public 0.1.25, scorer era tabench-1.1). Each row of the table above links to the run directories in the repository, where run.json, the answers, the verdicts and the transcript of every run can be read.