Most coding benchmarks ask one question: does the patch pass tests?
FrontierCode, from Cognition (the team behind Devin), asks a harder one: would the maintainer actually merge this PR?
That shift matches what I see on client repos. Agents can write code that turns green in CI and still gets rejected for scope creep, weak tests, or style that fights the house rules.
Why pass rate misleads buyers
SWE-bench and cousins grade against reference solutions or inherited test suites. METR's March 2026 note is worth reading: many SWE-bench-passing PRs would not be merged into main in the real project.
FrontierCode is built around maintainer judgment distilled into rubrics:
| Dimension | What reviewers care about |
|---|---|
| Correctness | Does the change actually solve the brief? |
| Test quality | Are new tests meaningful, not theater? |
| Scope discipline | Did the agent touch only what the task required? |
| Style | Does it read like this codebase, not a generic LLM patch? |
| Regression safety | Would you trust this in production? |
Cognition pairs maintainer-written briefs with an ensemble of graders: unit tests, rubrics, and new verifier types. They also restrict internet use. Models can read docs and search errors like an engineer, but runs that consult solution-bearing sources (including the original PR) get zeroed.
How the benchmark was built
This is not a scraped GitHub mining job. Cognition worked with maintainers from 36 flagship open-source repos. Each maintainer spent 40+ hours per task, iterating with Cognition researchers until the rubric captured what "mergeable" means in that repo.
Tasks ship in three nested tiers:
| Tier | Tasks | Role |
|---|---|---|
| Extended | 150 | Full set |
| Main | 100 hardest | Primary reporting tier in 1.1 |
| Diamond | 50 hardest | Launched June 2026, deprecated July 2026 |
Cognition reports each model at five trials per reasoning effort, averaged, then picks the best effort level per model. That reduces lucky one-shot wins.
The numbers that reset expectations
On Diamond at launch (June 2026), the leaderboard looked humbling:
| Model | Diamond score |
|---|---|
| Claude Opus 4.8 | 13.4% |
| GPT-5.5 | 6.3% |
| Gemini 3.1 Pro | 4.7% |
| Kimi K2.6 (best open) | 3.8% |
On Main, Opus 4.8 led at 34.3%. On Extended, 51.8%. Even the leader fails roughly half the time on the full set.
GPT-5.5 scored lower on raw performance but used up to 4× fewer tokens than Opus 4.8 in Cognition's runs. That cost-intelligence trade matters when you bill agent loops by token.
Cognition also claims 81% fewer grading errors than SWE-Bench Pro in their QC pipeline. Independent reproduction still matters, but the direction is right: inherited PR tests disagree with human judgment far more often than maintainer rubrics do.
FrontierCode leaderboardWhat I do with a 13% ceiling
I do not treat FrontierCode as a vendor ad for Devin. I treat it as a recalibration tool:
- Runnable code is table stakes. If your eval only checks green tests, you are optimizing the wrong bar.
- Human review stays on the critical path. A 13% Diamond score means even the best model still produces mostly rejectable PRs on the hardest maintainer standards.
- Scope discipline is the silent killer. Agents love drive-by refactors. FrontierCode punishes that explicitly.
My practical harness stack:
- Use public benchmarks (SWE-bench Verified, DeepSWE) for coarse model routing.
- Add merge-style rubrics on 5–10 internal tickets before you trust an agent on a client monorepo.
- Log reviewer reasons (scope, tests, style) the same way FrontierCode does, so you can see whether failures cluster.
If you are comparing agents for a team, read DeepSWE's spread on long-horizon tasks alongside FrontierCode. DeepSWE breaks ties on hard engineering work. FrontierCode tells you whether the output is shippable, not just executable.
FrontierCode 1.1 and Diamond deprecation
In July 2026 Cognition shipped FrontierCode 1.1 with tighter internet-use rules and deprecated Diamond. Hard-task solve rates were so low that Diamond scores became noisy, and the subset no longer matched the true hardest 50 tasks after methodology updates.
Going forward, Main and Extended are the tiers Cognition reports. The June headline ("13 out of 100") still matters historically. It captured the gap between demo culture and maintainer culture at peak difficulty.
Takeaway for applied AI shipping
Agents that pass tests but fail review are not production-ready. FrontierCode is the first public eval I have seen that grades that gap directly.
If you are standing up agent evals for a real codebase and want help translating public benchmarks into internal scorecards, book a free discovery call.