Benchmark
MMLU-Pro
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating…
- Released
- 2024-06-03
- Modality
- text
- Categories
- legal, math, reasoning, language, finance, general, healthcare
- Openness
- open
- Source
- llm_stats
- Reported scores
- 134
Reported scores
llm_stats
| Model | Organization | Reported value | Reported | Evidence |
|---|---|---|---|---|
| Claude 3 Opus | Anthropic | 0.685 | 2024-02-29 | evidence |
| Claude 3 Sonnet | Anthropic | 0.568 | 2024-02-29 | evidence |
| Claude 3.5 Haiku | Anthropic | 0.65 | 2024-10-22 | evidence |
| Claude 3.5 Sonnet | Anthropic | 0.761 | 2024-06-21 | evidence |
| Claude 3.5 Sonnet | Anthropic | 0.776 | 2024-10-22 | evidence |
| DeepSeek-R1-0528 | DeepSeek | 0.85 | 2025-05-28 | evidence |
| DeepSeek-V3 | DeepSeek | 0.759 | 2024-12-25 | evidence |
| DeepSeek-V3 0324 | DeepSeek | 0.812 | 2025-03-25 | evidence |
| DeepSeek-V3.1 | DeepSeek | 0.837 | 2025-01-10 | evidence |
| DeepSeek-V3.2 | DeepSeek | 0.85 | 2025-12-01 | evidence |
| DeepSeek-V3.2 (Thinking) | DeepSeek | 0.85 | 2025-12-01 | evidence |
| DeepSeek-V3.2-Exp | DeepSeek | 0.85 | 2025-09-29 | evidence |
| DeepSeek-V4-Flash-0423 | DeepSeek | 0.864 | 2026-04-23 | evidence |
| DeepSeek-V4-Flash-Max | DeepSeek | 0.862 | 2026-04-23 | evidence |
| DeepSeek-V4-Pro-Max | DeepSeek | 0.875 | 2026-04-23 | evidence |
| DiffusionGemma 26B-A4B | 0.776 | 2026-06-10 | evidence | |
| ERNIE 4.5 | Baidu | 0.16 | 2025-06-25 | evidence |
| ERNIE 5.0 | Baidu | 0.87 | 2026-01-22 | evidence |
| Gemini 1.5 Flash | 0.673 | 2024-05-01 | evidence | |
| Gemini 1.5 Flash 8B | 0.587 | 2024-03-15 | evidence | |
| Gemini 1.5 Pro | 0.758 | 2024-05-01 | evidence | |
| Gemini 2.0 Flash | 0.764 | 2024-12-01 | evidence | |
| Gemini 2.0 Flash-Lite | 0.716 | 2025-02-05 | evidence | |
| Gemma 3 12B | 0.606 | 2025-03-12 | evidence | |
| Gemma 3 1B | 0.147 | 2025-03-12 | evidence | |
| Gemma 3 27B | 0.675 | 2025-03-12 | evidence | |
| Gemma 3 4B | 0.436 | 2025-03-12 | evidence | |
| Gemma 3n E2B Instructed | 0.405 | 2025-06-26 | evidence | |
| Gemma 3n E2B Instructed LiteRT (Preview) | 0.405 | 2025-05-20 | evidence | |
| Gemma 3n E4B Instructed | 0.506 | 2025-06-26 | evidence | |
| Gemma 3n E4B Instructed LiteRT Preview | 0.506 | 2025-05-20 | evidence | |
| Gemma 4 12B | 0.772 | 2026-05-23 | evidence | |
| Gemma 4 26B-A4B | 0.826 | 2026-04-02 | evidence | |
| Gemma 4 31B | 0.852 | 2026-04-02 | evidence | |
| Gemma 4 E2B | 0.6 | 2026-04-02 | evidence | |
| Gemma 4 E4B | 0.694 | 2026-04-02 | evidence | |
| GLM-4.5 | Z.ai | 0.846 | 2025-07-28 | evidence |
| GLM-4.5-Air | Z.ai | 0.814 | 2025-07-28 | evidence |
| GLM-4.7 | Z.ai | 0.843 | 2025-12-22 | evidence |
| GPT OSS 120B High | OpenAI | 0.807 | 2025-08-05 | evidence |
| GPT-4o | OpenAI | 0.726 | 2024-05-13 | evidence |
| GPT-4o | OpenAI | 0.747 | 2024-08-06 | evidence |
| Grok-1.5 | xAI | 0.51 | 2024-03-28 | evidence |
| Grok-2 | xAI | 0.755 | 2024-08-13 | evidence |
| Grok-2 mini | xAI | 0.72 | 2024-08-13 | evidence |
| Hermes 3 70B | Nous Research | 0.4724 | 2024-08-15 | evidence |
| Jamba 1.5 Large | AI21 Labs | 0.535 | 2024-08-22 | evidence |
| Jamba 1.5 Mini | AI21 Labs | 0.425 | 2024-08-22 | evidence |
| K-EXAONE-236B-A23B | LG AI Research | 0.838 | 2025-12-31 | evidence |
| Kimi K2 0905 | Moonshot AI | 0.825 | 2025-09-05 | evidence |
| Kimi K2 Base | Moonshot AI | 0.692 | 2025-07-11 | evidence |
| Kimi K2 Instruct | Moonshot AI | 0.811 | 2025-07-11 | evidence |
| Kimi K2-Instruct-0905 | Moonshot AI | 0.811 | 2025-09-05 | evidence |
| Kimi K2-Thinking-0905 | Moonshot AI | 0.846 | 2025-09-05 | evidence |
| Kimi K2.5 | Moonshot AI | 0.871 | 2026-01-27 | evidence |
| Llama 3.1 405B Instruct | Meta | 0.733 | 2024-07-23 | evidence |
| Llama 3.1 70B Instruct | Meta | 0.664 | 2024-07-23 | evidence |
| Llama 3.1 8B Instruct | Meta | 0.483 | 2024-07-23 | evidence |
| Llama 3.3 70B Instruct | Meta | 0.689 | 2024-12-06 | evidence |
| Llama 4 Maverick | Meta | 0.805 | 2025-04-05 | evidence |
| Llama 4 Scout | Meta | 0.743 | 2025-04-05 | evidence |
| LongCat-Flash-Chat | Meituan | 0.8268 | 2025-08-29 | evidence |
| LongCat-Flash-Lite | Meituan | 0.7829 | 2026-02-05 | evidence |
| LongCat-Flash-Thinking | Meituan | 0.826 | 2025-09-22 | evidence |
| MAI-Thinking-1 | Microsoft | 0.85 | 2026-06-02 | evidence |
| MiMo-V2-Flash | Xiaomi | 0.849 | 2025-12-16 | evidence |
| MiMo-V2.5-Pro | Xiaomi | 0.685 | 2026-04-27 | evidence |
| MiniCPM-SALA | OpenBMB | 0.6704 | 2026-02-11 | evidence |
| MiniMax M1 40K | MiniMax | 0.806 | 2025-06-16 | evidence |
| MiniMax M1 80K | MiniMax | 0.811 | 2025-06-16 | evidence |
| MiniMax M2 | MiniMax | 0.82 | 2025-10-27 | evidence |
| MiniMax M2.1 | MiniMax | 0.88 | 2025-12-23 | evidence |
| Mistral Small 3 24B Base | Mistral | 0.5437 | 2025-01-30 | evidence |
| Mistral Small 3 24B Instruct | Mistral | 0.663 | 2025-01-30 | evidence |
| Mistral Small 3.1 24B Base | Mistral | 0.5603 | 2025-03-17 | evidence |
| Mistral Small 3.1 24B Instruct | Mistral | 0.6676 | 2025-03-17 | evidence |
| Mistral Small 3.2 24B Instruct | Mistral | 0.6906 | 2025-06-20 | evidence |
| Mistral Small 4 | Mistral | 0.78 | 2026-03-16 | evidence |
| Nemotron 3 Nano (30B A3B) | NVIDIA | 0.783 | 2025-12-15 | evidence |
| Nemotron 3 Super (120B A12B) | NVIDIA | 0.8373 | 2026-03-11 | evidence |
| Nemotron 3 Ultra (550B A55B) | NVIDIA | 0.868 | 2026-06-04 | evidence |
| Nemotron 3.5 Lightning (30B A3B) | NVIDIA | 0.8194 | 2026-08-11 | evidence |
| North Micro Vision Instruct | Cohere | 0.307 | 2026-08-12 | evidence |
| Nova 2 Lite | Amazon | 0.809 | 2025-12-02 | evidence |
| Nova 2 Omni | Amazon | 0.807 | 2025-12-02 | evidence |
| Nova 2 Pro | Amazon | 0.816 | 2025-12-02 | evidence |
| Phi 4 | Microsoft | 0.704 | 2024-12-12 | evidence |
| Phi 4 Mini | Microsoft | 0.528 | 2025-02-01 | evidence |
| Phi 4 Reasoning | Microsoft | 0.743 | 2025-04-30 | evidence |
| Phi 4 Reasoning Plus | Microsoft | 0.76 | 2025-04-30 | evidence |
| Phi-3.5-mini-instruct | Microsoft | 0.474 | 2024-08-23 | evidence |
| Phi-3.5-MoE-instruct | Microsoft | 0.453 | 2024-08-23 | evidence |
| Qwen2 72B Instruct | Qwen | 0.644 | 2024-07-23 | evidence |
| Qwen2 7B Instruct | Qwen | 0.441 | 2024-07-23 | evidence |
| Qwen2.5 14B Instruct | Qwen | 0.637 | 2024-09-19 | evidence |
| Qwen2.5 32B Instruct | Qwen | 0.69 | 2024-09-19 | evidence |
| Qwen2.5 72B Instruct | Qwen | 0.711 | 2024-09-19 | evidence |
| Qwen2.5 7B Instruct | Qwen | 0.563 | 2024-09-19 | evidence |
| Qwen2.5 VL 32B Instruct | Qwen | 0.688 | 2025-02-28 | evidence |
| Qwen2.5-Coder 32B Instruct | Qwen | 0.504 | 2024-09-19 | evidence |
| Qwen2.5-Coder 7B Instruct | Qwen | 0.401 | 2024-09-19 | evidence |
| Qwen2.5-Omni-7B | Qwen | 0.47 | 2025-03-27 | evidence |
| Qwen3 235B A22B | Qwen | 0.6818 | 2025-04-29 | evidence |
| Qwen3 VL 235B A22B Instruct | Qwen | 0.818 | 2025-09-22 | evidence |
| Qwen3 VL 235B A22B Thinking | Qwen | 0.838 | 2025-09-22 | evidence |
| Qwen3 VL 30B A3B Instruct | Qwen | 0.778 | 2025-09-22 | evidence |
| Qwen3 VL 30B A3B Thinking | Qwen | 0.805 | 2025-09-22 | evidence |
| Qwen3 VL 32B Instruct | Qwen | 0.786 | 2025-09-22 | evidence |
| Qwen3 VL 32B Thinking | Qwen | 0.821 | 2025-09-22 | evidence |
| Qwen3 VL 4B Instruct | Qwen | 0.671 | 2025-09-22 | evidence |
| Qwen3 VL 4B Thinking | Qwen | 0.736 | 2025-09-22 | evidence |
| Qwen3 VL 8B Instruct | Qwen | 0.716 | 2025-09-22 | evidence |
| Qwen3 VL 8B Thinking | Qwen | 0.773 | 2025-09-22 | evidence |
| Qwen3-235B-A22B-Instruct-2507 | Qwen | 0.83 | 2025-07-22 | evidence |
| Qwen3-235B-A22B-Thinking-2507 | Qwen | 0.844 | 2025-07-25 | evidence |
| Qwen3-Next-80B-A3B-Instruct | Qwen | 0.806 | 2025-09-10 | evidence |
| Qwen3-Next-80B-A3B-Thinking | Qwen | 0.827 | 2025-09-10 | evidence |
| Qwen3.5-0.8B | Qwen | 0.423 | 2026-03-02 | evidence |
| Qwen3.5-122B-A10B | Qwen | 0.867 | 2026-02-24 | evidence |
| Qwen3.5-27B | Qwen | 0.861 | 2026-02-24 | evidence |
| Qwen3.5-2B | Qwen | 0.665 | 2026-03-02 | evidence |
| Qwen3.5-35B-A3B | Qwen | 0.853 | 2026-02-24 | evidence |
| Qwen3.5-397B-A17B | Qwen | 0.878 | 2026-02-16 | evidence |
| Qwen3.5-4B | Qwen | 0.791 | 2026-03-02 | evidence |
| Qwen3.5-9B | Qwen | 0.825 | 2026-03-02 | evidence |
| Qwen3.6 Plus | Qwen | 0.885 | 2026-04-02 | evidence |
| Qwen3.6-27B | Qwen | 0.862 | 2026-04-21 | evidence |
| Qwen3.6-35B-A3B | Qwen | 0.852 | 2026-04-16 | evidence |
| Qwen3.7 Max | Qwen | 0.896 | 2026-05-19 | evidence |
| Qwen3.7-Plus | Qwen | 0.885 | 2026-05-31 | evidence |
| Sakana Namazu | Sakana AI | 0.9033 | 2026-08-03 | evidence |
| Sarvam-105B | Sarvam AI | 0.817 | 2026-03-06 | evidence |
| Sarvam-30B | Sarvam AI | 0.8 | 2026-03-06 | evidence |
| Solar Pro 4 | Upstage | 0.863 | 2026-08-06 | evidence |
Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.