real-world@v2
The 5-tier difficulty ladder over issues we already fixed, replayed on forks of the real repos frozen at the pre-fix commit. The task is the original symptom report; the checker is the real fix's tests plus hidden assertions at factoring-neutral boundaries; tier weights 1/2/4/8/16 make the gradient and the headline is the derived "solves up to tier N" rung.
Difficulty ladder
Every task sits on a difficulty tier, every subject runs every tier, and tier weights make the gradient: the capability score is dominated by the highest tiers cleared. The headline is the derived "solves up to tier N" - the highest rung with a solved task.
| Tier | Tasks | Weight per task | What sits here |
|---|---|---|---|
| 1 | 2 | 1 | a localized single-file bug; the symptom names the surface it appears on |
| 2 | 3 | 2 | a single-service bug, symptom one hop from the cause |
| 3 | 2 | 4 | cause in a different module than the symptom; multi-file fix |
| 4 | 3 | 8 | cross-boundary: a contract mismatch between services, a fix spanning conventions |
| 5 | 2 | 16 | problems that took a human hours: races, misleading symptoms, fixes needing a design choice |
Headroom is designed, not broken. A top-tier task that no current agent solves is kept as long as its reference solution passes calibration: it is reachable and proven solvable, and its emptiness is the finding. The sample task below is deliberately a low-tier task - high-tier fixtures are production code and never publishable.
Grade bands
| Grade | Capability ≥ |
|---|---|
| A | 0.9 |
| B | 0.75 |
| C | 0.55 |
| D | 0.35 |
| F | below D |
This suite is marked outputs-reveal-prompts: its non-sample tasks publish scores and metadata only, because the outputs would otherwise reconstruct the prompts.
Sample task
Published in full, exactly as the models see it. All other tasks in the suite stay private.
No sample task published.