real-world@v3
The difficulty ladder, re-authored so the top rung separates a strong harness from grok-4.5 (electricitybench-171, after v2 saturated). Same fork-of-a-real-repo-at-the-pre-fix-commit format as v2; the task is the original symptom report and the checker is the real fix's tests plus hidden assertions; tier weights 1/2/4/8/16 make the gradient and the headline is the derived "solves up to tier N" rung.
Difficulty ladder
Every task sits on a difficulty tier, every subject runs every tier, and tier weights make the gradient: the capability score is dominated by the highest tiers cleared. The headline is the derived "solves up to tier N" - the highest rung with a solved task.
| Tier | Tasks | Weight per task | What sits here |
|---|---|---|---|
| 1 | 2 | 1 | a localized single-file bug; the symptom names the surface it appears on |
| 2 | 3 | 2 | a single-service bug, symptom one hop from the cause |
| 3 | 2 | 4 | cause in a different module than the symptom; multi-file fix |
| 4 | 3 | 8 | cross-boundary: a contract mismatch between services, a fix spanning conventions |
| 5 | 5 | 16 | problems that took a human hours: races, misleading symptoms, fixes needing a design choice |
Headroom is designed, not broken. A top-tier task that no current agent solves is kept as long as its reference solution passes calibration: it is reachable and proven solvable, and its emptiness is the finding.
Grade bands
| Grade | Capability ≥ |
|---|---|
| A | 0.9 |
| B | 0.75 |
| C | 0.55 |
| D | 0.35 |
| F | below D |
This suite is marked outputs-reveal-prompts: its tasks publish scores and metadata only, because the outputs would otherwise reconstruct the prompts.
Task privacy
Every task in this suite stays private - publishing task content would let it leak into training data and grade memorization instead of capability. What is public is the method: the ladder shape, weights, scoring, and grade bands above, plus every graded run's scores and operational metadata.