Benchmark
DeepSWE 1.1 Benchmark
What is DeepSWE 1.1?
DeepSWE 1.1 evaluates software engineering agents using the mini-swe-agent harness. Benchmark Radar tracks 25 reported scores for DeepSWE 1.1, cited by 1…
- Modality
- text
- Evaluates
- agents, code
- Openness
- unknown
- Importer
- LLM Stats
- Reported scores
- 25
Source provenance
- Original evidence https://api.zeroeval.com/leaderboard/benchmarks/deepswe-1.1?top_n=500
DeepSWE 1.1 paper and code
DeepSWE 1.1 results and reported scores
LLM Stats
| Model | Organization | Reported value | Reported | Evidence |
|---|---|---|---|---|
| Claude Fable 5 | Anthropic | 0.7 | 2026-06-09 | evidence |
| Claude Opus 4.8 | Anthropic | 0.59 | 2026-05-28 | evidence |
| Claude Opus 5 | Anthropic | 0.688 | 2026-07-24 | evidence |
| Claude Sonnet 4.6 | Anthropic | 0.3 | 2026-02-17 | evidence |
| Claude Sonnet 5 | Anthropic | 0.54 | 2026-06-30 | evidence |
| Gemini 3.1 Pro | 0.12 | 2026-02-19 | evidence | |
| Gemini 3.5 Flash | 0.37 | 2026-05-19 | evidence | |
| Gemini 3.6 Flash | 0.49 | 2026-07-21 | evidence | |
| Gemini 3.7 Flash | 0.653 | 2026-08-13 | evidence | |
| GLM-5.2 | Z.ai | 0.44 | 2026-06-16 | evidence |
| GLM-5.3 | Z.ai | 0.669 | 2026-08-14 | evidence |
| GPT-5.4 | OpenAI | 0.52 | 2026-03-05 | evidence |
| GPT-5.5 | OpenAI | 0.67 | 2026-04-23 | evidence |
| GPT-5.6 Luna | OpenAI | 0.67 | 2026-07-09 | evidence |
| GPT-5.6 Sol | OpenAI | 0.73 | 2026-07-09 | evidence |
| GPT-5.6 Terra | OpenAI | 0.7 | 2026-07-09 | evidence |
| Grok 4.5 | xAI | 0.54 | 2026-07-16 | evidence |
| Grok 4.6 | xAI | 0.659 | 2026-08-12 | evidence |
| Kimi K2.7 Code | Moonshot AI | 0.31 | 2026-06-12 | evidence |
| Kimi K3 | Moonshot AI | 0.69 | 2026-07-16 | evidence |
| Laguna S 2.1 | Poolside | 0.404 | 2026-07-21 | evidence |
| Muse Spark 1.1 | Meta | 0.53 | 2026-07-09 | evidence |
| Muse Spark 1.2 | Meta | 0.593 | 2026-08-05 | evidence |
| Qwen3.8 Max | Qwen | 0.566 | 2026-08-02 | evidence |
| Qwen3.8-27B | Qwen | 0.422 | 2026-08-14 | evidence |
Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.