Most public leaderboards aggregate the same handful of synthetic benchmarks that labs optimize for. We take a different approach focused on real-world utility. We author our own task suites from production codebases, keep them private, run them against coding agents on standard subscriptions, and publish the graded results.
Private suites, public grades
Keeping tasks private prevents models from overfitting or training against the evaluation set. The tasks remain behind the curtain so they measure genuine problem-solving ability, while the graded outcomes, capability scores, task durations, and plan usage are published openly.
Graded as complete agents on real plans
We evaluate complete coding agents - the harness, model, configuration, and subscription plan together - exactly as an engineer would use them. Cost is measured in subscription allowance consumed per run on the agent's actual plan tier, alongside median completion time and capability tiers.
What you can explore
- Follow the active standings on the Leaderboard.
- Inspect our evaluation framework and test suites on the Methodology page.
- Track agent progress over time on Trends.
- Compare subscription value with the Subscription Quiz.