Spec planning
spec-planning@v1 - the suite key used in the data and the URL
Given a real repository and a large feature request, write an implementation spec - graded on whether it foresaw the problems the real implementation actually hit. Four shipped features, 25 hidden checks each, mined from the fix trails.
scoring
judge
repeats
1
judge
grok-cli-grok-4-5
Grade bands
| Grade | Capability ≥ |
|---|---|
| A+ | 96.6667% |
| A | 93.3333% |
| A- | 90% |
| B+ | 85% |
| B | 80% |
| B- | 75% |
| C+ | 68.3333% |
| C | 61.6667% |
| C- | 55% |
| D+ | 48.3333% |
| D | 41.6667% |
| D- | 35% |
| F | below the lowest passing band |
This suite is marked outputs-reveal-prompts: its tasks publish scores and metadata only, because the outputs would otherwise reconstruct the prompts.
Task privacy
Every task in this suite stays private - publishing task content would let it leak into training data and grade memorization instead of capability. What is public is the method: the ladder shape, weights, scoring, and grade bands above, plus every graded run's scores and operational metadata.