Benchmark
DeepSWE Benchmark
What is DeepSWE?
DeepSWE is a software engineering agent benchmark evaluated with the mini-swe-agent harness, where each task is solved in an isolated container with no…
- Modality
- text
- Evaluates
- agents, code
- Openness
- unknown
- Importer
- LLM Stats
- Reported scores
- 11
Source provenance
- Original evidence https://api.zeroeval.com/leaderboard/benchmarks/deepswe?top_n=500
DeepSWE results and reported scores
LLM Stats
| Model | Organization | Reported value | Reported | Evidence |
|---|---|---|---|---|
| DeepSeek-V4-Flash-0731 | DeepSeek | 0.544 | 2026-07-31 | evidence |
| DeepSeek-V4-Pro-0813 | DeepSeek | 0.627 | 2026-08-13 | evidence |
| GLM-5.2 | Z.ai | 0.462 | 2026-06-16 | evidence |
| GPT-5.6 Luna | OpenAI | 0.672 | 2026-07-09 | evidence |
| GPT-5.6 Sol | OpenAI | 0.727 | 2026-07-09 | evidence |
| GPT-5.6 Terra | OpenAI | 0.696 | 2026-07-09 | evidence |
| Grok 4.5 | xAI | 0.53 | 2026-07-16 | evidence |
| Hy3 | Tencent | 0.28 | 2026-07-06 | evidence |
| Kimi K3 | Moonshot AI | 0.675 | 2026-07-16 | evidence |
| Seed 2.1 Pro | ByteDance | 0.327 | 2026-06-24 | evidence |
| Seed 2.1 Turbo | ByteDance | 0.23 | 2026-06-24 | evidence |
Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.