Editor's note (12 August 2026)
This post is from the bench's first week and is kept as written; everything below this note is the July text, not current practice. Three things have changed. The subject we grade is the whole coding agent - harness, model, settings and subscription plan as one entry - not a raw model behind an API. Cost is the share of that agent's own plan a run consumes, read from the provider's meter where there is one, never a per-call dollar figure, and two plans' shares are never ranked against each other as money. And no suite publishes a task's full input and output: task text stays private, and an output is published only where it can be separated from the private suite safely. The current method is on the methodology page.
A grade on Electricity Bench is not a single headline percentage. It is a per-suite roll-up designed to survive contact with reality.
Private suites, public grades
Each suite is a set of tasks we author and keep private. Keeping the tasks private is the whole point: a benchmark that a lab can see is a benchmark a lab can optimize for, and once that happens the number stops measuring capability and starts measuring familiarity. We publish grades, metadata, and, for a small set of tasks marked as samples, the full input and output. Everything else stays sealed.
Capability, cost, and latency together
A model that solves a task for ten cents is not equivalent to one that solves it for ten dollars, and a grade that ignores that is telling you half the story. Every suite cell carries the grade alongside the median cost and the latency it took to get there. You can compare capability, but you can also see what that capability costs.
Comparable only within a suite version
Grades are comparable within a suite version and nothing more. When we change a suite we bump its version and start a fresh column; we never quietly re-scale old numbers to fit new ones. Results are append-only. Corrections are new records, not edits.
Read the full methodology on the suites page.