This week we are adding a new suite on the bench: supervisor. It mimics how we currently work with how good the models got as orchestrators. An agent can now work as a supervisor and manage subagents and coordination and parallelizes the work. Our suite contains a single run of five real features from one of our repos and the agent works as a lead with a team of subagents.
The five features are real. They come from a service we run, frozen at the commit before we built them. They are specced including what should change and what must not. The supervisor's instruction is the one we give our own sessions: a subagent per feature, in parallel, each owning its files, one commit per feature, and the test suite should be green at the end.
Workers are pinned to a cheaper model of the same family, enforced by the harness. Fable supervises Opus, Opus supervises Sonnet, Astra supervises Sol, Sol supervises Luna, and Grok supervises Grok. This is what a supervisor setup looks like in practice as using the top model burns too much quota.
Grading is per task and includes hidden tests for the fixes we shipped after the initial implementation. A red test suite at the end scores zero, whatever the individual tasks say, because a supervisor who leaves the tree broken has landed nothing. The score is the share of the board that landed.
The main thing about this suite is not the complexity of the tasks. It is how much time and usage a supervised run takes. In calibration two very different setups each landed four of the five features in about twenty minutes, and missed the same one. What separates them is what it cost in the plan's window and how long it took, and that is what we will be measuring week over week.
First results land with next week's weekly run.