Benchmark
MMLU
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and…
- Modality
- text
- Categories
- legal, math, reasoning, language, finance, general, healthcare
- Openness
- unknown
- Source
- llm_stats
- Reported scores
- 101
Reported scores
llm_stats
| Model | Organization | Reported value | Reported | Evidence |
|---|---|---|---|---|
| Claude 3 Haiku | Anthropic | 0.752 | 2024-03-13 | evidence |
| Claude 3 Opus | Anthropic | 0.868 | 2024-02-29 | evidence |
| Claude 3 Sonnet | Anthropic | 0.79 | 2024-02-29 | evidence |
| Claude 3.5 Sonnet | Anthropic | 0.904 | 2024-06-21 | evidence |
| Claude 3.5 Sonnet | Anthropic | 0.904 | 2024-10-22 | evidence |
| Command R+ | Cohere | 0.757 | 2024-08-30 | evidence |
| DeepSeek-V2.5 | DeepSeek | 0.804 | 2024-05-08 | evidence |
| DeepSeek-V3 | DeepSeek | 0.885 | 2024-12-25 | evidence |
| ERNIE 4.5 | Baidu | 0.419 | 2025-06-25 | evidence |
| Gemini 1.0 Pro | 0.718 | 2024-02-15 | evidence | |
| Gemini 1.5 Flash | 0.789 | 2024-05-01 | evidence | |
| Gemini 1.5 Pro | 0.859 | 2024-05-01 | evidence | |
| Gemma 2 27B | 0.752 | 2024-06-27 | evidence | |
| Gemma 2 9B | 0.713 | 2024-06-27 | evidence | |
| Gemma 3n E2B Instructed | 0.601 | 2025-06-26 | evidence | |
| Gemma 3n E2B Instructed LiteRT (Preview) | 0.601 | 2025-05-20 | evidence | |
| Gemma 3n E4B Instructed | 0.649 | 2025-06-26 | evidence | |
| Gemma 3n E4B Instructed LiteRT Preview | 0.649 | 2025-05-20 | evidence | |
| GPT OSS 120B | OpenAI | 0.9 | 2025-08-05 | evidence |
| GPT OSS 20B | OpenAI | 0.853 | 2025-08-05 | evidence |
| GPT-3.5 Turbo | OpenAI | 0.698 | 2023-03-21 | evidence |
| GPT-4 | OpenAI | 0.864 | 2023-06-13 | evidence |
| GPT-4 Turbo | OpenAI | 0.865 | 2024-04-09 | evidence |
| GPT-4.1 | OpenAI | 0.902 | 2025-04-14 | evidence |
| GPT-4.1 mini | OpenAI | 0.875 | 2025-04-14 | evidence |
| GPT-4.1 nano | OpenAI | 0.801 | 2025-04-14 | evidence |
| GPT-4.5 | OpenAI | 0.908 | 2025-02-27 | evidence |
| GPT-4o | OpenAI | 0.887 | 2024-05-13 | evidence |
| GPT-4o | OpenAI | 0.857 | 2024-08-06 | evidence |
| GPT-4o mini | OpenAI | 0.82 | 2024-07-18 | evidence |
| GPT-5 | OpenAI | 0.925 | 2025-08-07 | evidence |
| Granite 3.3 8B Base | IBM | 0.6389 | 2025-04-16 | evidence |
| Granite 3.3 8B Instruct | IBM | 0.6554 | 2025-04-16 | evidence |
| Grok-1.5 | xAI | 0.813 | 2024-03-28 | evidence |
| Grok-2 | xAI | 0.875 | 2024-08-13 | evidence |
| Grok-2 mini | xAI | 0.862 | 2024-08-13 | evidence |
| Hermes 3 70B | Nous Research | 0.7909 | 2024-08-15 | evidence |
| IBM Granite 4.0 Tiny Preview | IBM | 0.604 | 2025-05-02 | evidence |
| Jamba 1.5 Large | AI21 Labs | 0.812 | 2024-08-22 | evidence |
| Jamba 1.5 Mini | AI21 Labs | 0.697 | 2024-08-22 | evidence |
| Kimi K2 0905 | Moonshot AI | 0.902 | 2025-09-05 | evidence |
| Kimi K2 Base | Moonshot AI | 0.878 | 2025-07-11 | evidence |
| Kimi K2 Instruct | Moonshot AI | 0.895 | 2025-07-11 | evidence |
| Kimi K2-Instruct-0905 | Moonshot AI | 0.895 | 2025-09-05 | evidence |
| Kimi-k1.5 | Moonshot AI | 0.874 | 2025-01-20 | evidence |
| Llama 3.1 405B Instruct | Meta | 0.873 | 2024-07-23 | evidence |
| Llama 3.1 70B Instruct | Meta | 0.836 | 2024-07-23 | evidence |
| Llama 3.1 8B Instruct | Meta | 0.694 | 2024-07-23 | evidence |
| Llama 3.1 Nemotron 70B Instruct | NVIDIA | 0.802 | 2024-10-01 | evidence |
| Llama 3.2 11B Instruct | Meta | 0.73 | 2024-09-25 | evidence |
| Llama 3.2 3B Instruct | Meta | 0.634 | 2024-09-25 | evidence |
| Llama 3.2 90B Instruct | Meta | 0.86 | 2024-09-25 | evidence |
| Llama 3.3 70B Instruct | Meta | 0.86 | 2024-12-06 | evidence |
| Llama 4 Maverick | Meta | 0.855 | 2025-04-05 | evidence |
| Llama 4 Scout | Meta | 0.796 | 2025-04-05 | evidence |
| LongCat-Flash-Chat | Meituan | 0.8971 | 2025-08-29 | evidence |
| LongCat-Flash-Lite | Meituan | 0.8552 | 2026-02-05 | evidence |
| MiMo-V2.5-Pro | Xiaomi | 0.894 | 2026-04-27 | evidence |
| Ministral 3 (14B Base 2512) | Mistral | 0.794 | 2025-12-04 | evidence |
| Ministral 3 (3B Base 2512) | Mistral | 0.707 | 2025-12-04 | evidence |
| Ministral 3 (8B Base 2512) | Mistral | 0.761 | 2025-12-04 | evidence |
| Ministral 8B Instruct | Mistral | 0.65 | 2024-10-16 | evidence |
| Mistral Large 2 | Mistral | 0.84 | 2024-07-24 | evidence |
| Mistral NeMo Instruct | Mistral | 0.68 | 2024-07-18 | evidence |
| Mistral Small 3 24B Base | Mistral | 0.8073 | 2025-01-30 | evidence |
| Mistral Small 3.1 24B Base | Mistral | 0.8101 | 2025-03-17 | evidence |
| Mistral Small 3.1 24B Instruct | Mistral | 0.8062 | 2025-03-17 | evidence |
| Mistral Small 3.2 24B Instruct | Mistral | 0.805 | 2025-06-20 | evidence |
| North Micro Vision Instruct | Cohere | 0.504 | 2026-08-12 | evidence |
| Nova Lite | Amazon | 0.805 | 2024-11-20 | evidence |
| Nova Micro | Amazon | 0.776 | 2024-11-20 | evidence |
| Nova Pro | Amazon | 0.859 | 2024-11-20 | evidence |
| o1 | OpenAI | 0.918 | 2024-12-17 | evidence |
| o1-mini | OpenAI | 0.852 | 2024-09-12 | evidence |
| o1-preview | OpenAI | 0.908 | 2024-09-12 | evidence |
| o3-mini | OpenAI | 0.869 | 2025-01-30 | evidence |
| Phi 4 | Microsoft | 0.848 | 2024-12-12 | evidence |
| Phi 4 Mini | Microsoft | 0.673 | 2025-02-01 | evidence |
| Phi-3.5-mini-instruct | Microsoft | 0.69 | 2024-08-23 | evidence |
| Phi-3.5-MoE-instruct | Microsoft | 0.789 | 2024-08-23 | evidence |
| Pixtral-12B | Mistral | 0.692 | 2024-09-17 | evidence |
| Qwen2 72B Instruct | Qwen | 0.823 | 2024-07-23 | evidence |
| Qwen2 7B Instruct | Qwen | 0.705 | 2024-07-23 | evidence |
| Qwen2.5 14B Instruct | Qwen | 0.797 | 2024-09-19 | evidence |
| Qwen2.5 32B Instruct | Qwen | 0.833 | 2024-09-19 | evidence |
| Qwen2.5 VL 32B Instruct | Qwen | 0.784 | 2025-02-28 | evidence |
| Qwen2.5-Coder 32B Instruct | Qwen | 0.751 | 2024-09-19 | evidence |
| Qwen2.5-Coder 7B Instruct | Qwen | 0.676 | 2024-09-19 | evidence |
| Qwen3 235B A22B | Qwen | 0.8781 | 2025-04-29 | evidence |
| Qwen3 VL 235B A22B Instruct | Qwen | 0.888 | 2025-09-22 | evidence |
| Qwen3 VL 235B A22B Thinking | Qwen | 0.906 | 2025-09-22 | evidence |
| Qwen3 VL 30B A3B Instruct | Qwen | 0.85 | 2025-09-22 | evidence |
| Qwen3 VL 30B A3B Thinking | Qwen | 0.876 | 2025-09-22 | evidence |
| Qwen3 VL 32B Instruct | Qwen | 0.864 | 2025-09-22 | evidence |
| Qwen3 VL 32B Thinking | Qwen | 0.887 | 2025-09-22 | evidence |
| Qwen3 VL 4B Instruct | Qwen | 0.772 | 2025-09-22 | evidence |
| Qwen3 VL 4B Thinking | Qwen | 0.815 | 2025-09-22 | evidence |
| Qwen3 VL 8B Instruct | Qwen | 0.807 | 2025-09-22 | evidence |
| Qwen3 VL 8B Thinking | Qwen | 0.852 | 2025-09-22 | evidence |
| Sarvam-105B | Sarvam AI | 0.906 | 2026-03-06 | evidence |
| Sarvam-30B | Sarvam AI | 0.851 | 2026-03-06 | evidence |
Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.