Editor's note (12 August 2026)
Kept as written, from the bench's first week; everything below this note is the July text. What changed: calling a model directly on a metered API is no longer the graded path. The subject is the complete agent - it runs headless through its own harness and tools, on the subscription plan you would buy - so the harness is not a confound to strip out, it is part of what is being measured, and the figure under it is the share of the plan a run consumed rather than a per-call dollar cost. The append-only rule below still holds, and this note is an instance of it. Current method: methodology.
Publishing a fair number for a model is harder than it looks, and most of the difficulty is invisible in the final figure.
Metering has to be fair
A model called through an agent wrapper is not the same as the model called directly. The wrapper adds its own system prompt, retries, and tool scaffolding, and it reports cost and latency for the whole apparatus rather than the model. If you grade that, you are grading the harness. So we call each model on the same footing and record the per-call cost, token counts, and latency ourselves, through a budget guard that gates every paid request.
Third-party numbers are context, not grades
Provider-claimed scores and community rankings are useful, but they are not ours and they are not neutral. We carry them in a clearly marked context box on each scorecard, every one tagged with its source, retrieval date, and whether it is self-reported or independent. None of them are ever folded into the grade we assign.
Append-only, on purpose
A benchmark you can silently edit is a benchmark you cannot trust. Our records are append-only: a regrade is a new entry that supersedes an old one, with the history intact. If we get something wrong, you will be able to see both what we said and what we corrected it to.
This is slower than scraping a leaderboard. We think that is the right trade.