Methodology
The method is fully public; only the questions are private. That is what makes "tests private" credible rather than suspicious.
How grading works
Each suite is a set of graded tasks with a fixed scoring method - programmatic checks or a pinned LLM judge. Every agent runs each task once per weekly grading run, so a week's grade is one measured pass and not an average of retries.
The vibe-coding suite is the exception: it walks an adaptive descent, starting at the hardest level and stepping down until the agent succeeds, so the result is the level it reached.
Capability rolls up to a letter grade per suite via published grade bands, alongside plan cost, task time, and reliability. Grades are comparable within a suite version only.
Latest grading run: W40 2026. Every published week is listed under weekly runs.