Context tokens per run

How this figure is computed

Definition. For run r, context(r) = tokens_input(r) + tokens_cache_read(r), where tokens_input is the uncached tail of every request and tokens_cache_read the cached prefix re-read on each call. Both are summed over all API calls of the run, including sub-agent threads, from the per-call record named by token_source in the run record (a gateway ledger for TrueArchitect; the harness session logs or the harness's own per-call usage for the comparison group). Cache writes and output tokens are excluded here and reported in the run records.

Aggregation. A column is the arithmetic mean of context(r) over the valid, scored runs of one arm at one model, for the selected exam and battery. When both batteries are selected, each battery forms its own cell and the column is the mean of the two cell means, so the longer battery does not dominate.

Why per model. Token counts are tokenizer facts: a Sonnet token and a GPT token are not the same unit. The site therefore never pools raw token counts across vendors. The pooled-models view (the default) pools an arm over its roster only where the roster is a single vendor; an arm spanning two vendors stays split into one column per model whatever the columns choice, which the table discloses.

Reading it. Lower is better. The like-for-like comparison is column to column at the same model: TrueArchitect at Sonnet 5 against bare Claude Code at Sonnet 5, TrueArchitect at GPT 5.6 Sol against Codex at GPT 5.6 Sol. The dashed reference lines are TrueArchitect's and the bare harnesses' pooled means, drawn only where a pooled value is licensed (a single-vendor roster), so the indexing tools have both TrueArchitect and their baseline in the same picture.

Disclosures

Exam protocols have different token denominators (a MultiTurn question replays its history; ZeroShot pays the fixed prefix per question). Switch protocols with the control above; the site never pools across them.

A run whose token facts were not captured is excluded from its column, never counted as zero; the table's n is the number of runs behind each column.

Provenance

Computed at build time from epochs/E004/summary/runs.json of the proof package (export tool tabench export public 0.1.25, scorer era tabench-1.1). Each row of the table above links to the run directories in the repository, where run.json, the answers, the verdicts and the transcript of every run can be read.