Benchmark

MMMLU Benchmark

What is MMMLU?

A language-count average. Two cards reporting "MMMLU" may be averaging over different language sets, so the counts must match to compare.

Released: 2024-09-24
Evaluates: multilingual
Openness: unknown
Importer: Model reports
Reported scores: 5

Source provenance

Original evidence https://huggingface.co/datasets/openai/MMMLU

MMMLU links

Website https://huggingface.co/datasets/openai/MMMLU

MMMLU results and reported scores

Model reports

Model	Organization	Reported value	Reported	Evidence
Claude Opus 4	Anthropic	88.8	2025-05-22	evidence
Claude Sonnet 4	Anthropic	86.5	2025-05-22	evidence
Gemini 3.1 Pro	Google	92.6	2026-02-19	evidence
Gemma 4 (31B)	Google	88.4	2026-03-11	evidence
Qwen3.5-397B-A17B	Qwen	88.5	2026-02-16	evidence

Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.

Which sources cite MMMLU?

DeepSeek-V4-Pro DeepSeek, 2026-04-22
Gemma 4 (31B) Google, 2026-03-11
Gemini 3.1 Pro Google, 2026-02-19
Qwen3.5-397B-A17B Qwen, 2026-02-16
Mistral Large 3 (675B) Mistral, 2025-11-28
Claude Opus 4 and Sonnet 4 Anthropic, 2025-05-22
Llama 4 Meta, 2025-04-05

Related benchmarks

MMMLU