There is a new suite on the bench: spec-planning. It measures something none of our other suites do: can a coding agent look at a real codebase, read a big feature request, and see the problems coming - before writing a single line of code?
Every task drops the agent into one of our production repositories, frozen at the moment just before a major feature was built, with the feature request exactly as it would have been filed - the goal, not a checklist. The agent's job is to explore the code and write the implementation plan. Here is the trick: we already built that feature, and we kept the receipts. Every painful follow-up fix it needed became a hidden check - 25 per task - covering collisions with existing mechanisms, cross-service contracts, lifecycle and concurrency, data and migration, and everything else that had to change. A judge reads the plan and asks of each one: did the spec see this coming? The traps that took us longest to figure out ourselves count double, and vague caution earns nothing - "consider data migrations" scores zero, naming the actual key collision scores.
Why build this? Coding agents keep getting better at fixing well-described bugs - our execution suites keep converging at the top - but our own fix history says the expensive failures happen earlier, in the things the plan never anticipated. Early testing suggests foresight really is its own skill: a plan written with hindsight aces every check, a competent happy-path plan scores near zero, the strongest model we tried led on every single task, and agents that look nearly identical on execution suites split apart here. There is also plenty of open room above the best result we have seen - which is exactly what we want a benchmark to have.
The first spec-planning results land with next week's weekly run.