Publishing fair, dependable numbers for AI coding agents is harder than it looks, and most of the nuance is lost in single-number benchmarks.
Measuring real subscription quota
Evaluating raw foundation models via metered API tokens fails to reflect what engineers actually experience. Developers use coding agents through subscription plans (0/mo, 00/mo, 00/mo) that enforce sliding usage windows (e.g. 5-hour session caps or weekly allowances). How much work an agent completes before exhausting a session window is a critical operational metric. We run agents against dedicated subscription accounts and meter plan usage directly, giving developers realistic cost expectations.
Third-party numbers are context, not grades
Provider-claimed scores and community rankings provide helpful context, but they are not independent. We display external numbers (such as LMArena Elo snapshots or vendor launch claims) in clearly labeled context boxes on each scorecard, tagged with their source and retrieval date. These figures are never blended into our own grades or ranking algorithms.
Version discipline and honest baselines
A benchmark you can silently adjust over time is a benchmark you cannot trust. Grades on Electricity Bench are comparable strictly within the same suite version. When test suites or difficulty ladders are updated, we introduce a new version column rather than retroactively altering historical records.
Read our complete testing methodology on the Methodology page.