A grade on Electricity Bench is not a single aggregate percentage or a synthetic puzzle score. It is a multi-dimensional scorecard evaluating real-world capability, task execution duration, and subscription quota cost.
Private suites, sealed tasks
Each evaluation suite is a collection of challenging tasks authored from real production codebases and kept strictly private. Keeping the tasks private is essential: a benchmark that AI labs can inspect is a benchmark their training pipelines will optimize for. When tasks leak into training data, the benchmark stops measuring genuine coding capability and starts measuring memorization. We publish the graded outcomes, capability tiers, completion times, and usage metrics, while the underlying tasks remain sealed.
The complete agent as the subject
We evaluate the complete agent apparatus rather than a raw foundation model behind an isolated API. An agent operates through its native harness (Claude Code, Codex CLI, Grok CLI, Antigravity CLI, Kimi Code), utilizing its own tool-calling loops, terminal access, and editing workflows. The agent runs on the standard subscription plan you would actually buy, with its default settings and declared reasoning effort.
Multi-dimensional evaluation
A model that solves a task quickly while consuming modest quota is not equivalent to one that requires an hour and exhausts your entire daily allowance. Every scorecard measures three core dimensions together:
- Capability: The proportion of tasks correctly solved and the difficulty tier cleared.
- Task time: The median wall-clock time required for the agent to boot, navigate the codebase, edit files, and pass test assertions.
- Plan quota: The percentage of the subscription allowance (e.g. 5-hour window or weekly cap) consumed by a benchmark run.
Comparable only within a suite version
Grades are strictly comparable only within the same suite version. Whenever a suite evolves - adding new issues or refining test cases - we bump its version and start a fresh comparison baseline. We never retroactively scale or adjust historical numbers to fit newer evaluations.
Explore our active suites and test design on the Methodology page.