We ran the full DeepSWE benchmark three ways. Same model, same scaffold, same real issues from real open source repositories. The only difference is what the agent can ask about the codebase before it edits.
GitNexus wins on all four measures at once: it solves more while spending less, writing fewer tokens and taking fewer steps.
| Setup | Pass rate | $ per trial | Output tokens | Steps |
|---|---|---|---|---|
| GitNexus | 68.37% | $0.6008 | 21,077 | 44.94 |
| Graphify | 54.02% | $0.6364 | 21,769 | 48.00 |
| Bare model | 36.99% | $0.6631 | 22,252 | 50.25 |
The usual trade is more accuracy for more spend. Here both move the same way, which works out to $0.88 per solved task against $1.79.
| Setup | Pass rate | $ per trial | Output tokens | Steps | $ per solved task |
|---|---|---|---|---|---|
| GitNexus | 68.37% | $0.6008 | 21,077 | 44.94 | $0.88 |
| Graphify | 54.02% | $0.6364 | 21,769 | 48.00 | $1.18 |
| Bare model | 36.99% | $0.6631 | 22,252 | 50.25 | $1.79 |
| Averages across 3,471 trials, 10 per task per arm. 95% CI on pass rate: GitNexus 65.6%–71.0%, bare model 34.3%–39.8%. | |||||
Without a code graph, an agent navigates by reading. That searching is what the last three numbers measure.
The graph answers what calls this and what depends on it, so the agent stops opening files to find out.
Steps drop from 50.2 to 44.9A resolved answer is far smaller than the pile of files an agent would otherwise re-read every turn.
Input tokens 15.8% fewerGrounded edits need less speculative reasoning, so output shrinks while the success rate climbs.
Output 5.3% fewer, pass rate 1.85×When a task is hard, navigation is the bottleneck rather than the edit itself.
3.5× the pass rate on the hardest 58 tasksThe lead is not carried by a few tasks. GitNexus comes out ahead on 98 of the 113, and the gap grows as the work gets harder.
On the 58 hardest tasks the bare model manages 15.9% and GitNexus manages 55.9%. Those are the tasks where understanding the repository is the real obstacle.
| Difficulty | Tasks | GitNexus | Graphify | Bare model | Advantage |
|---|---|---|---|---|---|
| Easy · bare model solves over 66% | 17 | 94.7% | 78.9% | 80.8% | +13.9 pts |
| Moderate · bare model solves 33 to 66% | 38 | 77.3% | 66.6% | 49.9% | +27.4 pts |
| Hard · bare model solves under 33% | 58 | 55.9% | 39.1% | 15.9% | +40.0 pts |
| Per task head to head: GitNexus wins 98, ties 8, loses 7 against the bare model. Two proportion z test on pass rate: z = 15.11, p < 1e-50. | |||||
DeepSWE is a set of real bugs and feature requests taken from open source projects. Every task is an issue someone filed, paired with the test suite that was failing at the time. The manifest holds 116 entries spread across 89 different projects, covering Python, Go, TypeScript and Rust. The agent has to find its way around an unfamiliar codebase before it can fix anything. That is the part the code graph changes.
Single-issue benchmarks measure one narrow thing: how an agent reads code. They cannot measure querying hundreds of repositories as one graph, blast radius before a merge, or taint tracking across service boundaries. Those capabilities are the product. This is what happens on the narrow slice anyway.
Want the raw trajectories, or a run on your own repositories?