Benchmark

MMLU Benchmark

What is MMLU?

Saturated at the frontier and heavily contaminated. Useful as a historical baseline, not as a ranking signal. Benchmark Radar tracks 4 reported scores for…

Released: 2020-09-07
Evaluates: knowledge
Openness: unknown
Importer: Model reports
Reported scores: 4

Source provenance

Original evidence https://github.com/hendrycks/test

MMLU code

Code https://github.com/hendrycks/test

MMLU results and reported scores

Model reports

Model	Organization	Reported value	Reported	Evidence
DeepSeek-R1	DeepSeek	90.8	2025-01-22	evidence
DeepSeek-V3	DeepSeek	88.5	2024-12-27	evidence
Gemini 1.5 Pro	Google	85.9	2024-03-08	evidence
Mistral Large 2	Mistral	84.0	2024-07-24	evidence

Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.

Which sources cite MMLU?

DeepSeek-V4-Pro DeepSeek, 2026-04-22
Olmo 3 (7B and 32B) Ai2, 2025-11-20
GPT-4.1 OpenAI, 2025-04-14
DeepSeek-R1 DeepSeek, 2025-01-22
DeepSeek-V3 DeepSeek, 2024-12-27
Qwen2.5-Coder Qwen, 2024-09-18
Mistral Large 2 Mistral, 2024-07-24
Llama 3.1 405B Meta, 2024-07-23
Gemini 1.5 Google, 2024-03-08