codex-cli + gpt-5.6-luna
codex-cli harness · ChatGPT Plus
overall pending Two distinct suite families are required for an overall grade.
Per-suite results
Run on codex-cli codex-cli 0.144.6, model gpt-5.6-luna, Jul 19, 2026.
One run of this suite consumes 0.1% of ChatGPT Plus - about 1685.4 runs per week. Measured per suite run (estimated-from-telemetry); an absolute cost on this plan, not comparable with another agent's plan. Measured in a declared quiet window on a shared subscription: operator ChatGPT Plus login shared between eb-runner and the claude host; graded runs only inside the scheduled quiet window, all recurring tasks blocked for its duration.
Trajectory
Every scored run of this agent, oldest first. Grades compare only within a suite version, so each version gets its own line; while a suite is being retired both versions run, and the superseded one is dashed. The filled point is the run the scorecard shows today. All agents over time →
Jul 19, 2026: A on v1
Compare
- codex-cli + gpt-5.6-luna vs Claude Sonnet 5 (CLI)
- codex-cli + gpt-5.6-luna vs claude-code + claude-fable-5
- codex-cli + gpt-5.6-luna vs claude-code + claude-haiku-4-5
- codex-cli + gpt-5.6-luna vs claude-code + claude-opus-4-8
- codex-cli + gpt-5.6-luna vs claude-code + claude-sonnet-5
- codex-cli + gpt-5.6-luna vs codex-cli + gpt-5.6-sol
- codex-cli + gpt-5.6-luna vs codex-cli + gpt-5.6-terra
- codex-cli + gpt-5.6-luna vs cursor-cli + composer-2.5
- codex-cli + gpt-5.6-luna vs grok-cli + grok-4.5
- codex-cli + gpt-5.6-luna vs kimi-cli + kimi-k3
Transcript excerpts
Outputs are shown only when they can be separated safely from the private suite. Scores and run metadata remain visible when an output is withheld.
| Task | # | Turns | Latency | Output |
|---|---|---|---|---|
| a1b2c3 | 1 | - | 8473 ms | {"invoice_number":"INV-4471","total":135.5,"currency":"USD","line_items":[{"description":"Widget assembly","amount":120} |
| a3f7c8 | 1 | - | 5270 ms | output only{"doc_id":"PO-2024-0817","issued_date":"2024-08-17","name":"Norman Vasquez","amount":3780.00,"currency":"USD"} |
| a6f1d9 | 1 | - | 10621 ms | output only{"claims":[{"claim":"The battery lasts 12 hours on a single charge.","metric":12,"unit":"hours"},{"claim":"It weighs jus |
| a9d4f2 | 1 | - | 5297 ms | output only{"incident_start":"2026-06-18T14:32:07+02:00","severity":"high","affected_component":"payment-gateway","error_count":3," |
| b1d7c3 | 1 | - | 5231 ms | output only{"records":[{"id":101,"price_usd":13,"qty":4},{"id":102,"price_usd":9,"qty":12},{"id":103,"price_usd":5.25,"qty":10},{"i |
| b2c9e1 | 1 | - | 5822 ms | output only{"merchant":"Bluebird Roastery","currency":"EUR","subtotal":12.7,"tax":1.27,"total":13.97,"line_items":[{"name":"Flat Wh |
| b8e3d7 | 1 | - | 6192 ms | output only{"product":"Torch Peak 400 Headlamp","specs":[{"name":"Max output","value":"400 lumens"},{"name":"Beam distance","value" |
| c3f8a4 | 1 | - | 5698 ms | output only{"contact_name":"Priya Ramanathan","contact_company":"Cedar Grove Logistics","contact_email":"priya.ramanathan@cedargrov |
| c4a9f1 | 1 | - | 6581 ms | output only{"product":"Cascade Pro 12 Espresso Machine","sku":"CP12-BLK-2048","specs":[{"name":"Pump pressure","value":"15 bar"},{" |
| c8e2a5 | 1 | - | 7445 ms | output only{"party_a":"Brightwave Analytics LLC","party_b":"Coppervale Logistics Inc.","start_date":"2025-03-03","term_months":18," |
| d1e8b6 | 1 | - | 5481 ms | output only{"incident_start":"2026-05-03T09:14:26Z","incident_end":"2026-05-03T09:14:39Z","primary_component":"inventory-service"," |
| d4e5f6 | 1 | - | 3565 ms | output only{"tracking":"1Z999AA10123456784","carrier":"UPS","delivered":true} |
| d6b2e9 | 1 | - | 3903 ms | output only{"reference_number":"MU-7741-C","effective_date":"2025-11-02","amount_due":1276.40,"account_holder":"Priya Raghunathan"} |
| d9c4b2 | 1 | - | 4391 ms | output only{"records":[{"id":1,"name":"Marlin Crate","amount":10.5},{"id":2,"name":"Harbor Bolt","amount":22},{"id":3,"name":"Tidew |
| e2b7c5 | 1 | - | 6471 ms | {"title":"Building Your First CLI in Rust","chapters":[{"title":"Introduction","start":"00:00","duration_seconds":105},{ |
| e4b9d1 | 1 | - | 6864 ms | withheldPrivate-suite content |
| e7b1c9 | 1 | - | 6379 ms | output only{"action_items":[{"task":"Update the onboarding deck with the new depot list","owner":"Marcus","due":"2026-07-20"},{"tas |
| f0a7d2 | 1 | - | 8297 ms | output only{"invoice_number":"INV-2025-0847","currency":"GBP","line_items":[{"description":"CNC machining service","quantity":12,"u |
| f3a8e6 | 1 | - | 5845 ms | output only{"records":[{"item":"Bolt Pack","weight_kg":0.85},{"item":"Steel Rod","weight_kg":2.5},{"item":"Copper Coil","weight_kg" |
| f5c2a8 | 1 | - | 8231 ms | output only{"product":"Nimbus Slate 9 Tablet","specs":[{"name":"Display","value":"10.9 inch LCD"},{"name":"Storage","value":"128 GB |