Electricity Bench
Menu

← Leaderboard

codex-cli + gpt-6-astra

codex-cli harness · ChatGPT Pro $100 ($100/mo)

COVERALL

Clears tier 2 of 5 (partial at tiers 3 and 5) on Real-world issues

Capability mean across 3 graded suite families.

capability
57%
plan cost per run
3.0% of ChatGPT Pro $100
median task time
1m 56s
reliability
100%
12345
What the tiers mean

"Clears" marks the highest rung where every task at it and below was solved. Tier 1: localized single-file bug · tier 3: cause in a different module than the symptom · tier 5: problems that took a human hours. A cleared rung solved every task on it; a partial rung solved some. The headline stops at the last unbroken rung, so a solved tier above a half-solved one counts only as partial.

Clears level 2 of 5 (partial at levels 3, 4 and 5) on Vibe coding v1 - the vaguest report level at which it still fixed every bug, having also cleared every richer level below
12345
What the levels mean

Level 1: a full brief naming the surface and the acceptance · level 3: the report as filed · level 5: a vague vibe report in a non-technical user's words. Context falls as the level rises, so vaguer is harder. A cleared level fixed every bug at it and below; a partial level fixed some. The headline stops at the last unbroken level, so a fix above a half-cleared level counts only as partial.

10 of this suite's tasks were credited rather than run: on a suite that asks the same problem at several levels of detail, solving it from the vaguest description credits the more detailed ones instead of asking again.

Per-suite results

C
Real-world issues v3
57%
plan cost per run
3.0% of ChatGPT Pro $100
median task time
1m 56s
reliability
100%
tier 1tier 2tier 3tier 4tier 5

11 of 15 tasks solved - grouped by tier, easiest first; hover a square for its time and turns.

Run on codex-cli codex-cli 0.153.4, model gpt-6-astra (-c model_reasoning_effort=medium), week 2026-W36.

One run of this suite consumes 3.0% of ChatGPT Pro $100.

How the plan cost was measured

Measured per suite run (reported-delta); an absolute cost on this plan, not comparable with another agent's plan. Measured in a declared quiet window on a shared subscription: RunByAI OpenAI contestant ChatGPT Pro $100 login, moved onto eb-runner on 2026-09-05 and still present on the claude host RunByAI pane; graded runs only inside the scheduled quiet window, all recurring tasks blocked for its duration.

C
Spec planning v1
74%
plan cost per run
1.0% of ChatGPT Pro $100
median task time
4m 0s
reliability
100%

0 of 4 tasks solved - hover a square for its time and turns.

Run on codex-cli codex-cli 0.153.4, model gpt-6-astra (-c model_reasoning_effort=medium), week 2026-W36.

One run of this suite consumes 1.0% of ChatGPT Pro $100.

How the plan cost was measured

Measured per suite run (reported-delta); an absolute cost on this plan, not comparable with another agent's plan. Measured in a declared quiet window on a shared subscription: RunByAI OpenAI contestant ChatGPT Pro $100 login, moved onto eb-runner on 2026-09-05 and still present on the claude host RunByAI pane; graded runs only inside the scheduled quiet window, all recurring tasks blocked for its duration.

C
Vibe coding v1
61%
plan cost per solve
0.2% of ChatGPT Pro $100
median task time
2m 54s
reliability
100%
level 5level 4level 3level 2level 1issue aissue bissue cissue dissue e

20 of 25 tasks solved (10 credited without running) - each row is one bug, vaguest report first and context growing to the right; hover a square for its time and turns.

Run on codex-cli codex-cli 0.153.4, model gpt-6-astra (-c model_reasoning_effort=medium), week 2026-W36.

One good solve on this suite consumes 0.2% of ChatGPT Pro $100 on average.

How the plan cost was measured

Counted over each issue's first solve (5 solves); the walk's failed harder cells and confirmation runs are the benchmark's own search cost and are excluded (full walk: 4.0% over 15 cells). Measured per cell run (reported-delta); an absolute cost on this plan, not comparable with another agent's plan. Measured in a declared quiet window on a shared subscription: RunByAI OpenAI contestant ChatGPT Pro $100 login, moved onto eb-runner on 2026-09-05 and still present on the claude host RunByAI pane; graded runs only inside the scheduled quiet window, all recurring tasks blocked for its duration.

Task outputs from these runs are withheld: publishing them would reveal the private suite. Per-task scores, times and turns are in the chips above.

Frequently asked questions

Is GPT-6 Astra worth it on ChatGPT Pro $100?

On our private real-world suite (real-world@v3), GPT-6 Astra earned grade C with 57% capability, clearing every problem up to tier 2 of 5 - a bug spanning a few files and landing some tier 5 problems. It runs on ChatGPT Pro $100 at $100/month (public price as of 2026-09-05). One full suite run consumed 3.0% of the plan's week window. Measured 2026-09-05.

How good is GPT-6 Astra at real-world coding?

GPT-6 Astra clears tier 2 of our five-tier real-world ladder - every task at that tier and below - where tier 1 is a localized single-file bug and tier 5 is a problem that took a human engineer hours. It solves some but not all tasks up at tier 5. Its capability score on real-world@v3 is 57% (grade C). Last benchmarked 2026-09-05.

Compare