Where every number comes from
The site is a lens over the repository. It is built from three files of the proof package and nothing else; the transcripts it links to are the repository's own files.
The files the site reads
| file | what it holds |
|---|---|
epochs/E004/summary/runs.json | one flat row per run: identity, validity, score and fail counts, the four token columns and their source, duration, LLM and tool call counts by class, index-query count, cost basis, path. Every figure is a function of this file. |
epochs/E004/summary/rate-tables.json | the USD-per-million-token tables behind every estimated cost, with the formula and the arm dispatch rule. |
epochs/E004/summary/per-question.csv | pass counts per arm × model × protocol × battery × question — Figure 10. |
epochs/E004/manifest.json | every run path with the SHA-256 of its record; the instrument freeze; container image digests; the scope predicate; the scorer era. |
repositories/…/batteries/*/questions.json | the questions with difficulty and categories (the grading key sits beside them). |
Reproducing a figure
Each figure page states its definition and aggregation rule. To recompute one: filter runs.json to the protocol and battery shown, keep rows with validity_ok = true and a non-null score, form cells of arm × battery × model, compute the statistic in each cell, and average cells with equal weight. The data table under every figure carries the per-column value, the number of runs and cells, the minimum, maximum and standard deviation, and links to the run directories.
The site's source is in the repository under site/; the build runs on every push and deploys the static output. Nothing is computed at request time.
Reproducing the package
The repository tree is the output of tabench export public --epoch N over the benchmark's store. manifest.json lists every run with the SHA-256 of its run.json, and each run.json lists the SHA-256 of every other file in its directory, so the tree is hash-chained from one file. The comparison group's container definitions are in docker-containers/ with the image digests each run carries; the TrueArchitect instrument is pinned as a freeze whose version record is in the manifest.
Export record
| epoch | E004 — Epoch 004 — line-4 |
|---|---|
| export tool | tabench export public 0.1.25 (schema public-run/1) |
| scorer era | tabench-1.1 |
| runs in scope | 2692 |
| TrueArchitect predicate | exam_epoch = N AND concept_arm_id = 'ta-ask' |
| comparison predicate | family IN (cc_bare, cc_competitor, harness_bare) AND no founding-import row AND not #pilot AND exam×battery pair present on the TA side |