automation-tool-use@v1
Drive synthetic tools correctly - workflow ordering, argument fidelity, error recovery, and refusing when no tool fits.
Table stakes. Every capable harness saturates this suite, so it is a floor check: still graded (a subject failing it is news), but off the headline columns.
scoring
programmatic
repeats
1
Grade bands
| Grade | Capability ≥ |
|---|---|
| A | 0.9 |
| B | 0.75 |
| C | 0.55 |
| D | 0.35 |
| F | below D |
Sample tasks
Published in full, exactly as the models see it. All other tasks in the suite stay private.
A customer, Dana Rowe (dana@northwind.example), asked for a 30-minute onboarding call on 2026-07-20. Book it on my calendar and make sure Dana gets an invite so it lands on her calendar too.
Please print the signed partnership contract, put it in an envelope, and post it by physical mail to our partner's head office at 4 Kingsway, London.