When every frontier model scores 55–58% on the same public coding benchmark, you do not have a leaderboard. You have a tie that tells you nothing about production.
DeepSWE, built by Datacurve, is the most convincing attempt I have seen this year to break that cluster. The AlphaSignal signal put GPT-5.5 at 70% with a 16-point gap to the next model. Even if exact numbers shift with v1.1 reruns, the spread is the story.
Why mined GitHub benchmarks mislead
Most agentic coding evals follow SWE-bench: pull merged fixes from public repos, grade with the tests that shipped with the PR.
That creates two structural bugs Datacurve calls out in their paper:
- Contamination: models may have seen the issue, discussion, and fix during training
- Noisy grading: inherited tests confirm one patch shape, not correct behavior
DeepSWE tasks are written from scratch across 91 active repos in TypeScript, Go, Python, JavaScript, and Rust. They are not contributed upstream, so reference solutions stay out of the public training scrape.
Each task uses a hand-written functional verifier that checks behavior, not implementation trivia. The paper reports an independent LLM judge disagrees with DeepSWE verifiers about 1.4% of the time versus 32.4% for SWE-Bench Pro inherited tests. That is a credibility gap worth paying attention to.
Harder tasks, shorter prompts
Counterintuitively, DeepSWE prompts are about half the length of SWE-Bench Pro prompts while reference solutions touch 5.5× more code and consume roughly 2× more output tokens.
That matches real engineering work better than "fix this tiny bug described in a novel." Agents must explore, implement, and integrate.
| Property | DeepSWE | Typical mined SWE eval |
|---|---|---|
| Task origin | Original, unreleased | Public merged PRs |
| Verifier | Hand-written behavior checks | Shipped unit tests |
| Code touched | High | Often small |
| Leaderboard spread | Wide at frontier | Compressed cluster |
How to run it yourself
Datacurve open-sourced the benchmark and drives evals through Pier, a Harbor-compatible runner with sandboxed agents.
Quickstart pattern:
git clone https://github.com/datacurve-ai/deep-swe uv tool install datacurve-pier export OPENAI_API_KEY=... pier run -p deep-swe/tasks --agent mini-swe-agent --model openai/gpt-5.5
Leaderboard scores use mini-swe-agent on Modal for consistency. If you swap in Claude Code or Codex CLI directly, treat results as directional, not comparable to the public board.
Useful links:
What I do with benchmarks like this
I do not pick models from screenshots. I pick them from task shapes:
- If your work is ticket-sized bug fixes, SWE-Bench Verified still signals something useful
- If your work is multi-file features with ambiguous specs, DeepSWE-style separation matters more
- If your bill is agent loops, pair benchmark gaps with cost per task (see agentic coding economics)
DeepSWE's GPT-5.5 lead aligns with OpenAI's Terminal-Bench and SWE-Bench Pro narrative for GPT-5.5. It also gives buyers a reason to pay for frontier models on hard repos instead of assuming "they all tie anyway."
The takeaway
Better benchmarks change purchasing decisions. DeepSWE is the first public coding eval in a while that made me rethink default model routing for long-horizon agent tasks.
If you are standing up an agent eval harness for your own codebase and want help translating public benchmarks into internal scorecards, book a free discovery call.