We cut a new private suite today: vibe-coding. It grades something none of our other suites isolate: how well a coding agent fixes a real bug as the bug report gets worse.
Every task in the suite is a real, already-fixed bug from our own production services, checked out at the commit just before the fix, with the repository's own tests plus hidden assertions grading the outcome. That part is the same format as our real-world suite. The difference is the axis. Real-world tiers the issue - how far the cause sits from the symptom. Vibe-coding tiers the prompt: the same five bugs, each asked at five levels of context, from a full engineering brief that names the surface and the acceptance criteria down to a vague complaint in a non-technical user's words. Twenty-five cells; within a row only the prompt changes. The headline per agent reads "solves down to context level N", with the vaguest level weighted heaviest.
The question behind it came out of our own calibration work: when an agent solves a hard bug from a great report, how much of that was the agent, and how much was the report? Now we can answer that per model - and what calibration showed is that the answer is not what general-coding scores predict. Rich briefs get solved by every reference we ran. The crack comes at the level where the report stops spelling out the one property an honest fix must also preserve - the retry path that must keep working, the second subsystem the repair touches - and different agents crack at genuinely different levels. Reading thin reports well turns out to be its own skill.
Two design notes for the methodology-minded. First, we run the grid adaptively: hardest cell first, adding context only downward, so cost scales with how much the grid still has to ask. A lone pass at a vague level does not get an agent credit for the whole row - crediting requires the next-easier cell to confirm it, because a single attempt at the crack boundary can go either way and we would rather pay for one confirmation run than publish a coin flip. Second, a graded attempt is one run: an agent gets exactly one attempt per cell, never a best-of-three. Our confidence comes from calibration instead - before the suite went live, every failing score from the reference agents was read back from its transcript to confirm the failure was the agent's, not a broken test or a garbled prompt. Those reviews caught three faults on our side; each was fixed and re-measured, and the failures that survived are all real.
First grades land with the next weekly run. Scorecards will grow a vibe-coding line alongside real-world, and the weekly writeup will read the two axes together.