Cost per correct answer
Cost per correct answer divides what a run cost by the number of questions it answered correctly: the vendor-reported figure where the harness reported one, otherwise the benchmark's estimate from the published rate tables over the run's four token columns.
Pooled models shows one column per arm, its models averaged with equal weight — the readable overview. Per model shows one column per model for every arm, for like-for-like reads at a fixed model. This choice applies to every figure on the site and is remembered.
Data behind this figure — ZeroShot + MultiTurn · both batteries
| arm | role | column | USD per correct answer | runs | cells | min | max | sd | rank | runs in the repository |
|---|---|---|---|---|---|---|---|---|---|---|
| TrueArchitect | TrueArchitect | 9 models | $0.115 | 330 | 34 | $0.004 | $0.424 | $0.086 | 2nd of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 · GPT 5.6 Luna · GPT 5.6 Terra · GPT 5.6 Sol |
| CC Bare | bare harness | 6 models | $0.310 | 201 | 22 | $0.071 | $0.939 | $0.186 | 6th of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| Codex Bare | bare harness | 3 models | $0.093 | 120 | 12 | $0.004 | $0.240 | $0.076 | 1st of 9 | GPT 5.6 Luna · GPT 5.6 Terra · GPT 5.6 Sol |
| Cursor Bare | bare harness | 8 models | $0.172 | 320 | 32 | $0.005 | $0.652 | $0.153 | 3rd of 9 | Haiku 4.5 · Sonnet 5 · Opus 5 · GPT 5.6 Luna · GPT 5.6 Terra · GPT 5.6 Sol · Grok 4.6 · Composer 2.5 |
| Graphify | indexing tool on Claude Code | 6 models | $0.342 | 201 | 22 | $0.065 | $0.901 | $0.199 | 9th of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| GitNexus | indexing tool on Claude Code | 6 models | $0.311 | 201 | 22 | $0.070 | $0.677 | $0.153 | 7th of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| CodeGraph | indexing tool on Claude Code | 6 models | $0.283 | 203 | 22 | $0.067 | $0.750 | $0.144 | 5th of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| CodebaseMemory | indexing tool on Claude Code | 6 models | $0.274 | 201 | 22 | $0.072 | $0.623 | $0.131 | 4th of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| Serena | indexing tool on Claude Code | 6 models | $0.338 | 207 | 23 | $0.071 | $0.898 | $0.189 | 8th of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
Figure 11. Cost per correct answer divides what a run cost by the number of questions it answered correctly: the vendor-reported figure where the harness reported one, otherwise the benchmark's estimate from the published rate tables over the run's four token columns. Each column is the mean over valid, scored runs at one model (or pooled over the arm's roster in the pooled-models view); TrueArchitect and the bare harnesses are the dashed references, and the data table names each row's cost basis. The context saving of Figure 1 becomes a price saving at every model, and the ordering survives the normalisation by correct answers.
How this figure is computed
Definition. For a scored run with score(r) > 0, cost_per_correct(r) = cost(r) / score(r), where cost(r) is the vendor-reported cost when the harness reported one (Claude Code's own result envelope) and otherwise the estimate Σ tokens × rate for the four token columns, using the rate table that applies to the arm: Cursor's published prices for the Cursor arm, Augment's vendor-list-plus-40-percent for Auggie, the vendor list prices for everything else. The tables are published as `epochs/E00N/summary/rate-tables.json`; each run row carries `cost_basis` naming which one priced it.
Aggregation. Cell means averaged with equal weight over the selected batteries, models and protocols. A run whose model has no rate row has no cost and is excluded from the mean, never counted as zero.
Reading it. Lower is better. Because estimates ride list prices, an arm on a subscription plan may pay less in practice; the figure is an API-equivalent price, comparable across arms because it is computed the same way for all of them. Codex reports no cache-write split, so its estimate charges written tokens at the input rate: a disclosed floor of at most 25 percent of the input component.
Disclosures
Vendor-reported and estimated costs sit in the same column; the table beneath the figure carries the basis per row and the rate tables are published beside the run rows.
Provenance
Computed at build time from epochs/E004/summary/runs.json of the proof package (export tool tabench export public 0.1.25, scorer era tabench-1.1). Each row of the table above links to the run directories in the repository, where run.json, the answers, the verdicts and the transcript of every run can be read.