Benchmark

MMMU Benchmark

What is MMMU?

MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and…

Released: 2023-11-27
Modality: multimodal
Evaluates: multimodal, reasoning, general, healthcare, vision
Openness: open
Importer: LLM Stats
Reported scores: 63

Source provenance

Original evidence https://api.zeroeval.com/leaderboard/benchmarks/mmmu?top_n=500

MMMU paper, code and dataset

Paper https://arxiv.org/abs/2311.16502
Code https://github.com/MMMU-Benchmark/MMMU
Dataset https://huggingface.co/datasets/MMMU/MMMU
Dataset https://huggingface.co/datasets/MMMU/MMMU_Pro

MMMU results and reported scores

LLM Stats

Model	Organization	Reported value	Reported	Evidence
Claude 3.5 Sonnet	Anthropic	0.683	2024-10-22	evidence
Claude 3.7 Sonnet	Anthropic	0.75	2025-02-24	evidence
Claude Sonnet 4	Anthropic	0.744	2025-05-22	evidence
Command A+	Cohere	0.751	2026-05-20	evidence
DeepSeek VL2	DeepSeek	0.511	2024-12-13	evidence
DeepSeek VL2 Small	DeepSeek	0.48	2024-12-13	evidence
DeepSeek VL2 Tiny	DeepSeek	0.407	2024-12-13	evidence
Gemini 1.0 Pro	Google	0.479	2024-02-15	evidence
Gemini 1.5 Flash	Google	0.623	2024-05-01	evidence
Gemini 1.5 Flash 8B	Google	0.537	2024-03-15	evidence
Gemini 1.5 Pro	Google	0.659	2024-05-01	evidence
Gemini 2.0 Flash	Google	0.707	2024-12-01	evidence
Gemini 2.0 Flash Thinking	Google	0.754	2025-01-21	evidence
Gemini 2.0 Flash-Lite	Google	0.68	2025-02-05	evidence
Gemini 2.5 Flash	Google	0.797	2025-05-20	evidence
Gemini 2.5 Flash-Lite	Google	0.729	2025-06-17	evidence
Gemini 2.5 Pro	Google	0.796	2025-05-20	evidence
Gemini 2.5 Pro Preview 06-05	Google	0.82	2025-06-05	evidence
GPT-3.5 Turbo	OpenAI	0.0	2023-03-21	evidence
GPT-4.1	OpenAI	0.748	2025-04-14	evidence
GPT-4.1 mini	OpenAI	0.727	2025-04-14	evidence
GPT-4.1 nano	OpenAI	0.554	2025-04-14	evidence
GPT-4.5	OpenAI	0.752	2025-02-27	evidence
GPT-4o	OpenAI	0.722	2024-08-06	evidence
GPT-4o mini	OpenAI	0.594	2024-07-18	evidence
GPT-5	OpenAI	0.842	2025-08-07	evidence
GPT-5.1	OpenAI	0.854	2025-11-13	evidence
GPT-5.1 Instant	OpenAI	0.854	2025-11-12	evidence
GPT-5.1 Thinking	OpenAI	0.854	2025-11-12	evidence
Grok-1.5	xAI	0.536	2024-03-28	evidence
Grok-1.5V	xAI	0.536	2024-04-12	evidence
Grok-2	xAI	0.661	2024-08-13	evidence
Grok-2 mini	xAI	0.632	2024-08-13	evidence
Grok-3	xAI	0.78	2025-02-17	evidence
Kimi-k1.5	Moonshot AI	0.7	2025-01-20	evidence
Llama 3.2 11B Instruct	Meta	0.507	2024-09-25	evidence
Llama 3.2 90B Instruct	Meta	0.603	2024-09-25	evidence
Llama 4 Maverick	Meta	0.734	2025-04-05	evidence
Llama 4 Scout	Meta	0.694	2025-04-05	evidence
Mistral Small 3.1 24B Base	Mistral	0.5927	2025-03-17	evidence
Mistral Small 3.1 24B Instruct	Mistral	0.5927	2025-03-17	evidence
Mistral Small 3.2 24B Instruct	Mistral	0.625	2025-06-20	evidence
Nova Lite	Amazon	0.562	2024-11-20	evidence
Nova Pro	Amazon	0.617	2024-11-20	evidence
o1	OpenAI	0.776	2024-12-17	evidence
o3	OpenAI	0.829	2025-04-16	evidence
o4-mini	OpenAI	0.816	2025-04-16	evidence
Phi-3.5-vision-instruct	Microsoft	0.43	2024-08-23	evidence
Phi-4-multimodal-instruct	Microsoft	0.551	2025-02-01	evidence
Pixtral Large	Mistral	0.64	2024-11-18	evidence
Pixtral-12B	Mistral	0.525	2024-09-17	evidence
QvQ-72B-Preview	Qwen	0.703	2024-12-25	evidence
Qwen2.5 VL 32B Instruct	Qwen	0.7	2025-02-28	evidence
Qwen2.5 VL 72B Instruct	Qwen	0.702	2025-01-26	evidence
Qwen2.5 VL 7B Instruct	Qwen	0.586	2025-01-26	evidence
Qwen2.5-Omni-7B	Qwen	0.592	2025-03-27	evidence
Qwen3.5-122B-A10B	Qwen	0.839	2026-02-24	evidence
Qwen3.5-27B	Qwen	0.823	2026-02-24	evidence
Qwen3.5-35B-A3B	Qwen	0.814	2026-02-24	evidence
Qwen3.6 Plus	Qwen	0.86	2026-04-02	evidence
Qwen3.6-27B	Qwen	0.829	2026-04-21	evidence
Qwen3.6-35B-A3B	Qwen	0.817	2026-04-16	evidence
Step3-VL-10B	StepFun	0.781	2026-01-15	evidence

Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.

Which sources cite MMMU?

MMMU