Benchmark

SWE-Bench Verified

A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to…

Modality
text
Categories
reasoning, frontend_development, code
Openness
unknown
Source
llm_stats
Reported scores
111

Reported scores

llm_stats

ModelOrganizationReported valueReportedEvidence
Claude 3.5 HaikuAnthropic0.4062024-10-22evidence
Claude 3.5 SonnetAnthropic0.492024-10-22evidence
Claude 3.7 SonnetAnthropic0.7032025-02-24evidence
Claude Fable 5Anthropic0.952026-06-09evidence
Claude Haiku 4.5Anthropic0.7332025-10-15evidence
Claude Mythos PreviewAnthropic0.9392026-04-07evidence
Claude Opus 4Anthropic0.7252025-05-22evidence
Claude Opus 4.1Anthropic0.7452025-08-05evidence
Claude Opus 4.5Anthropic0.8092025-11-24evidence
Claude Opus 4.6Anthropic0.8082026-02-05evidence
Claude Opus 4.7Anthropic0.8762026-04-16evidence
Claude Opus 4.8Anthropic0.8862026-05-28evidence
Claude Sonnet 4Anthropic0.7272025-05-22evidence
Claude Sonnet 4.6Anthropic0.7962026-02-17evidence
Claude Sonnet 5Anthropic0.8522026-06-30evidence
DeepSeek-R1-0528DeepSeek0.4462025-05-28evidence
DeepSeek-V2.5DeepSeek0.1682024-05-08evidence
DeepSeek-V3DeepSeek0.422024-12-25evidence
DeepSeek-V3.1DeepSeek0.662025-01-10evidence
DeepSeek-V3.2DeepSeek0.7312025-12-01evidence
DeepSeek-V3.2 (Thinking)DeepSeek0.7312025-12-01evidence
DeepSeek-V3.2-ExpDeepSeek0.6782025-09-29evidence
DeepSeek-V3.2-SpecialeDeepSeek0.7312025-12-01evidence
DeepSeek-V4-Flash-0423DeepSeek0.7862026-04-23evidence
DeepSeek-V4-Flash-MaxDeepSeek0.792026-04-23evidence
DeepSeek-V4-Pro-MaxDeepSeek0.8062026-04-23evidence
Devstral MediumMistral0.6162025-07-10evidence
Devstral Small 1.1Mistral0.5362025-07-11evidence
Gemini 2.5 FlashGoogle0.6042025-05-20evidence
Gemini 2.5 Flash-LiteGoogle0.3162025-06-17evidence
Gemini 2.5 ProGoogle0.6322025-05-20evidence
Gemini 2.5 Pro Preview 06-05Google0.6722025-06-05evidence
Gemini 3 FlashGoogle0.782025-12-17evidence
Gemini 3 ProGoogle0.7622025-11-18evidence
Gemini 3.1 ProGoogle0.8062026-02-19evidence
Gemini DiffusionGoogle0.2292025-05-20evidence
GLM-4.5Z.ai0.6422025-07-28evidence
GLM-4.5-AirZ.ai0.5762025-07-28evidence
GLM-4.6Z.ai0.682025-09-30evidence
GLM-4.7Z.ai0.7382025-12-22evidence
GLM-4.7-FlashZ.ai0.5922026-01-19evidence
GLM-5Z.ai0.7782026-02-11evidence
GPT-4.1OpenAI0.5462025-04-14evidence
GPT-4.1 miniOpenAI0.2362025-04-14evidence
GPT-4.5OpenAI0.382025-02-27evidence
GPT-4oOpenAI0.3322024-08-06evidence
GPT-4o miniOpenAI0.0872024-07-18evidence
GPT-5OpenAI0.7492025-08-07evidence
GPT-5 CodexOpenAI0.7452025-09-15evidence
GPT-5.1OpenAI0.7632025-11-13evidence
GPT-5.1 CodexOpenAI0.7372025-11-19evidence
GPT-5.1 InstantOpenAI0.7632025-11-12evidence
GPT-5.1 ThinkingOpenAI0.7632025-11-12evidence
GPT-5.2OpenAI0.82025-12-11evidence
Grok Code Fast 1xAI0.7082025-08-28evidence
Hy3Tencent0.782026-07-06evidence
Inkling-SmallThinking Machines Lab0.8022026-07-30evidence
Kimi K2-Instruct-0905Moonshot AI0.6582025-09-05evidence
Kimi K2-Thinking-0905Moonshot AI0.7132025-09-05evidence
Kimi K2.5Moonshot AI0.7682026-01-27evidence
Kimi K2.6Moonshot AI0.8022026-04-20evidence
Laguna XS 2.1Poolside0.7092026-07-02evidence
LongCat-Flash-ChatMeituan0.6042025-08-29evidence
LongCat-Flash-LiteMeituan0.5442026-02-05evidence
LongCat-Flash-ThinkingMeituan0.5942025-09-22evidence
LongCat-Flash-Thinking-2601Meituan0.72026-01-14evidence
MAI-Code-1-FlashMicrosoft0.7162026-06-02evidence
MAI-Code-1.1-FlashMicrosoft0.7262026-08-11evidence
MAI-Thinking-1Microsoft0.7352026-06-02evidence
MiMo-V2-FlashXiaomi0.7342025-12-16evidence
MiMo-V2-OmniXiaomi0.7482026-03-18evidence
MiMo-V2-ProXiaomi0.782026-03-18evidence
MiMo-V2.5-ProXiaomi0.7892026-04-27evidence
MiniMax M1 40KMiniMax0.5562025-06-16evidence
MiniMax M1 80KMiniMax0.562025-06-16evidence
MiniMax M2MiniMax0.6942025-10-27evidence
MiniMax M2.1MiniMax0.672025-12-23evidence
MiniMax M2.5MiniMax0.8022026-02-12evidence
MiniMax M3MiniMax0.8052026-06-01evidence
Mistral Medium 3.5Mistral0.7762026-04-29evidence
Muse Glimmer-30BMeta0.762026-08-10evidence
Muse SparkMeta0.7742026-04-08evidence
Nemotron 3 Nano (30B A3B)NVIDIA0.3882025-12-15evidence
Nemotron 3 Super (120B A12B)NVIDIA0.53732026-03-11evidence
Nemotron 3 Ultra (550B A55B)NVIDIA0.7072026-06-04evidence
Nemotron 3.5 Lightning (30B A3B)NVIDIA0.51562026-08-11evidence
North Mini Code 1.0Cohere0.6762026-06-09evidence
Nova 2 LiteAmazon0.6452025-12-02evidence
Nova 2 ProAmazon0.72025-12-02evidence
o1OpenAI0.412024-12-17evidence
o1-previewOpenAI0.4132024-09-12evidence
o3OpenAI0.6912025-04-16evidence
o3-miniOpenAI0.4932025-01-30evidence
o4-miniOpenAI0.6812025-04-16evidence
Qwen3 MaxQwen0.6962025-12-15evidence
Qwen3-Coder 480B A35B InstructQwen0.6962025-01-31evidence
Qwen3.5-122B-A10BQwen0.722026-02-24evidence
Qwen3.5-27BQwen0.7242026-02-24evidence
Qwen3.5-35B-A3BQwen0.6922026-02-24evidence
Qwen3.5-397B-A17BQwen0.7642026-02-16evidence
Qwen3.6 PlusQwen0.7882026-04-02evidence
Qwen3.6-27BQwen0.7722026-04-21evidence
Qwen3.6-35B-A3BQwen0.7342026-04-16evidence
Qwen3.7 MaxQwen0.8042026-05-19evidence
Qwen3.7-PlusQwen0.7772026-05-31evidence
Sarvam-105BSarvam AI0.452026-03-06evidence
Sarvam-30BSarvam AI0.342026-03-06evidence
Seed 2.0 LiteByteDance0.7352026-02-14evidence
Seed 2.0 ProByteDance0.7652026-02-14evidence
Solar Pro 4Upstage0.7062026-08-06evidence
Step-3.5-FlashStepFun0.7442026-02-02evidence

Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.