Benchmark

GPQA

A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and…

Released
2023-11-20
Modality
text
Categories
physics, reasoning, general, biology, chemistry
Openness
restricted
Source
llm_stats
Reported scores
239

Reported scores

llm_stats

ModelOrganizationReported valueReportedEvidence
ChatGPT-4o LatestOpenAI0.842024-05-13evidence
Claude 3 HaikuAnthropic0.3332024-03-13evidence
Claude 3 OpusAnthropic0.5042024-02-29evidence
Claude 3 SonnetAnthropic0.4042024-02-29evidence
Claude 3.5 HaikuAnthropic0.4162024-10-22evidence
Claude 3.5 SonnetAnthropic0.5942024-06-21evidence
Claude 3.5 SonnetAnthropic0.6722024-10-22evidence
Claude 3.7 SonnetAnthropic0.8482025-02-24evidence
Claude Haiku 4.5Anthropic0.732025-10-15evidence
Claude Mythos PreviewAnthropic0.9462026-04-07evidence
Claude Opus 4Anthropic0.7962025-05-22evidence
Claude Opus 4.1Anthropic0.8092025-08-05evidence
Claude Opus 4.5Anthropic0.872025-11-24evidence
Claude Opus 4.6Anthropic0.9132026-02-05evidence
Claude Opus 4.7Anthropic0.9422026-04-16evidence
Claude Opus 4.8Anthropic0.9362026-05-28evidence
Claude Sonnet 4Anthropic0.7542025-05-22evidence
Claude Sonnet 4.5Anthropic0.8342025-09-29evidence
Claude Sonnet 4.6Anthropic0.8992026-02-17evidence
DeepSeek R1 Distill Llama 70BDeepSeek0.6522025-01-20evidence
DeepSeek R1 Distill Llama 8BDeepSeek0.492025-01-20evidence
DeepSeek R1 Distill Qwen 1.5BDeepSeek0.3382025-01-20evidence
DeepSeek R1 Distill Qwen 14BDeepSeek0.5912025-01-20evidence
DeepSeek R1 Distill Qwen 32BDeepSeek0.6212025-01-20evidence
DeepSeek R1 Distill Qwen 7BDeepSeek0.4912025-01-20evidence
DeepSeek R1 ZeroDeepSeek0.7332025-01-20evidence
DeepSeek-R1-0528DeepSeek0.812025-05-28evidence
DeepSeek-V3DeepSeek0.5912024-12-25evidence
DeepSeek-V3 0324DeepSeek0.6842025-03-25evidence
DeepSeek-V3.1DeepSeek0.7492025-01-10evidence
DeepSeek-V3.2DeepSeek0.8242025-12-01evidence
DeepSeek-V3.2 (Thinking)DeepSeek0.8242025-12-01evidence
DeepSeek-V3.2-ExpDeepSeek0.7992025-09-29evidence
DeepSeek-V4-Flash-0423DeepSeek0.8742026-04-23evidence
DeepSeek-V4-Flash-MaxDeepSeek0.8812026-04-23evidence
DeepSeek-V4-Pro-MaxDeepSeek0.9012026-04-23evidence
DiffusionGemma 26B-A4BGoogle0.7322026-06-10evidence
ERNIE 4.5Baidu0.742025-06-25evidence
ERNIE 5.0Baidu0.852026-01-22evidence
Gemini 1.0 ProGoogle0.2792024-02-15evidence
Gemini 1.5 FlashGoogle0.512024-05-01evidence
Gemini 1.5 Flash 8BGoogle0.3842024-03-15evidence
Gemini 1.5 ProGoogle0.5912024-05-01evidence
Gemini 2.0 FlashGoogle0.6212024-12-01evidence
Gemini 2.0 Flash ThinkingGoogle0.7422025-01-21evidence
Gemini 2.0 Flash-LiteGoogle0.5152025-02-05evidence
Gemini 2.5 FlashGoogle0.8282025-05-20evidence
Gemini 2.5 Flash-LiteGoogle0.6462025-06-17evidence
Gemini 2.5 ProGoogle0.832025-05-20evidence
Gemini 2.5 Pro Preview 06-05Google0.8642025-06-05evidence
Gemini 3 FlashGoogle0.9042025-12-17evidence
Gemini 3 ProGoogle0.9192025-11-18evidence
Gemini 3.1 Flash-LiteGoogle0.8692026-03-03evidence
Gemini 3.1 ProGoogle0.9432026-02-19evidence
Gemini DiffusionGoogle0.4042025-05-20evidence
Gemma 3 12BGoogle0.4092025-03-12evidence
Gemma 3 1BGoogle0.1922025-03-12evidence
Gemma 3 27BGoogle0.4242025-03-12evidence
Gemma 3 4BGoogle0.3082025-03-12evidence
Gemma 3n E2B InstructedGoogle0.2482025-06-26evidence
Gemma 3n E2B Instructed LiteRT (Preview)Google0.2482025-05-20evidence
Gemma 3n E4B InstructedGoogle0.2372025-06-26evidence
Gemma 3n E4B Instructed LiteRT PreviewGoogle0.2372025-05-20evidence
Gemma 4 12BGoogle0.7882026-05-23evidence
Gemma 4 26B-A4BGoogle0.8232026-04-02evidence
Gemma 4 31BGoogle0.8432026-04-02evidence
Gemma 4 E2BGoogle0.4342026-04-02evidence
Gemma 4 E4BGoogle0.5862026-04-02evidence
GLM-4.5Z.ai0.7912025-07-28evidence
GLM-4.5-AirZ.ai0.752025-07-28evidence
GLM-4.6Z.ai0.812025-09-30evidence
GLM-4.7Z.ai0.8572025-12-22evidence
GLM-4.7-FlashZ.ai0.7522026-01-19evidence
GLM-5.1Z.ai0.8622026-04-07evidence
GLM-5.2Z.ai0.9122026-06-16evidence
GPT OSS 120BOpenAI0.8012025-08-05evidence
GPT OSS 120B HighOpenAI0.8092025-08-05evidence
GPT OSS 20BOpenAI0.7152025-08-05evidence
GPT OSS 20B HighOpenAI0.7422025-08-05evidence
GPT-3.5 TurboOpenAI0.3082023-03-21evidence
GPT-4OpenAI0.3572023-06-13evidence
GPT-4 TurboOpenAI0.482024-04-09evidence
GPT-4.1OpenAI0.6632025-04-14evidence
GPT-4.1 miniOpenAI0.652025-04-14evidence
GPT-4.1 nanoOpenAI0.5032025-04-14evidence
GPT-4.5OpenAI0.6952025-02-27evidence
GPT-4oOpenAI0.5362024-05-13evidence
GPT-4oOpenAI0.7012024-08-06evidence
GPT-4o miniOpenAI0.4022024-07-18evidence
GPT-5OpenAI0.8572025-08-07evidence
GPT-5 HighOpenAI0.8732025-08-07evidence
GPT-5 MediumOpenAI0.8812025-08-07evidence
GPT-5 miniOpenAI0.8232025-08-07evidence
GPT-5 nanoOpenAI0.7122025-08-07evidence
GPT-5.1OpenAI0.8812025-11-13evidence
GPT-5.1 HighOpenAI0.8812025-11-12evidence
GPT-5.1 InstantOpenAI0.8812025-11-12evidence
GPT-5.1 ThinkingOpenAI0.8812025-11-12evidence
GPT-5.2OpenAI0.9242025-12-11evidence
GPT-5.2 ProOpenAI0.9322025-12-11evidence
GPT-5.4OpenAI0.9282026-03-05evidence
GPT-5.4 miniOpenAI0.882026-03-17evidence
GPT-5.4 nanoOpenAI0.8282026-03-17evidence
GPT-5.5OpenAI0.9362026-04-23evidence
GPT-5.5 InstantOpenAI0.8562026-05-05evidence
GPT-5.6 LunaOpenAI0.9232026-07-09evidence
GPT-5.6 SolOpenAI0.9462026-07-09evidence
GPT-5.6 TerraOpenAI0.9292026-07-09evidence
Grok 4 FastxAI0.8572025-08-28evidence
Grok 4.5xAI0.932026-07-16evidence
Grok-1.5xAI0.3592024-03-28evidence
Grok-2xAI0.562024-08-13evidence
Grok-2 minixAI0.512024-08-13evidence
Grok-3xAI0.8462025-02-17evidence
Grok-3 MinixAI0.842025-02-17evidence
Grok-4xAI0.8752025-07-09evidence
Grok-4 HeavyxAI0.8842025-07-09evidence
Hermes 3 70BNous Research0.6612024-08-15evidence
Hy3Tencent0.9042026-07-06evidence
Inkling-SmallThinking Machines Lab0.8952026-07-30evidence
Jamba 1.5 LargeAI21 Labs0.3692024-08-22evidence
Jamba 1.5 MiniAI21 Labs0.3232024-08-22evidence
Kimi K2 0905Moonshot AI0.7582025-09-05evidence
Kimi K2 BaseMoonshot AI0.4812025-07-11evidence
Kimi K2 InstructMoonshot AI0.7512025-07-11evidence
Kimi K2-Instruct-0905Moonshot AI0.7512025-09-05evidence
Kimi K2-Thinking-0905Moonshot AI0.8452025-09-05evidence
Kimi K2.5Moonshot AI0.8762026-01-27evidence
Kimi K2.6Moonshot AI0.9052026-04-20evidence
Kimi K3Moonshot AI0.9352026-07-16evidence
Llama 3.1 405B InstructMeta0.5072024-07-23evidence
Llama 3.1 70B InstructMeta0.4172024-07-23evidence
Llama 3.1 8B InstructMeta0.3042024-07-23evidence
Llama 3.1 Nemotron Nano 8B V1NVIDIA0.5412025-03-18evidence
Llama 3.1 Nemotron Ultra 253B v1NVIDIA0.76012025-04-07evidence
Llama 3.2 11B InstructMeta0.3282024-09-25evidence
Llama 3.2 3B InstructMeta0.3282024-09-25evidence
Llama 3.2 90B InstructMeta0.4672024-09-25evidence
Llama 3.3 70B InstructMeta0.5052024-12-06evidence
Llama 4 MaverickMeta0.6982025-04-05evidence
Llama 4 ScoutMeta0.5722025-04-05evidence
Llama-3.3 Nemotron Super 49B v1NVIDIA0.66672025-03-18evidence
LongCat-Flash-ChatMeituan0.73232025-08-29evidence
LongCat-Flash-LiteMeituan0.66782026-02-05evidence
LongCat-Flash-ThinkingMeituan0.8152025-09-22evidence
LongCat-Flash-Thinking-2601Meituan0.8052026-01-14evidence
Magistral MediumMistral0.7082025-06-10evidence
Magistral Small 2506Mistral0.68182025-06-10evidence
MAI-Code-1-FlashMicrosoft0.8462026-06-02evidence
MAI-Thinking-1Microsoft0.8422026-06-02evidence
Mercury 2Inception0.742026-02-24evidence
MiMo-V2-FlashXiaomi0.8372025-12-16evidence
MiMo-V2.5-ProXiaomi0.6672026-04-27evidence
Min istral 3 (3B Reasoning 2512)Mistral0.5342025-12-04evidence
MiniMax M1 40KMiniMax0.6922025-06-16evidence
MiniMax M1 80KMiniMax0.72025-06-16evidence
MiniMax M2MiniMax0.782025-10-27evidence
MiniMax M2.1MiniMax0.812025-12-23evidence
Ministral 3 (14B Reasoning 2512)Mistral0.7122025-12-04evidence
Ministral 3 (8B Reasoning 2512)Mistral0.6682025-12-04evidence
Mistral Large 3 (675B Base)Mistral0.4392025-12-04evidence
Mistral Large 3 (675B Instruct 2512 Eagle)Mistral0.4392025-12-04evidence
Mistral Large 3 (675B Instruct 2512 NVFP4)Mistral0.4392025-12-04evidence
Mistral Large 3 (675B Instruct 2512)Mistral0.4392025-12-04evidence
Mistral Small 3 24B BaseMistral0.34372025-01-30evidence
Mistral Small 3 24B InstructMistral0.4532025-01-30evidence
Mistral Small 3.1 24B BaseMistral0.3752025-03-17evidence
Mistral Small 3.1 24B InstructMistral0.45962025-03-17evidence
Mistral Small 3.2 24B InstructMistral0.46132025-06-20evidence
Mistral Small 4Mistral0.7122026-03-16evidence
Muse Glimmer-30BMeta0.8352026-08-10evidence
Muse SparkMeta0.8952026-04-08evidence
Nemotron 3 Nano (30B A3B)NVIDIA0.752025-12-15evidence
Nemotron 3 Super (120B A12B)NVIDIA0.8272026-03-11evidence
Nemotron 3 Ultra (550B A55B)NVIDIA0.872026-06-04evidence
Nemotron 3.5 Lightning (30B A3B)NVIDIA0.75442026-08-11evidence
Nemotron Nano 9B v2NVIDIA0.642025-08-18evidence
Nova 2 LiteAmazon0.7962025-12-02evidence
Nova 2 ProAmazon0.8142025-12-02evidence
Nova LiteAmazon0.422024-11-20evidence
Nova MicroAmazon0.42024-11-20evidence
Nova ProAmazon0.4692024-11-20evidence
o1OpenAI0.782024-12-17evidence
o1-miniOpenAI0.62024-09-12evidence
o1-previewOpenAI0.7332024-09-12evidence
o1-proOpenAI0.792024-12-17evidence
o3OpenAI0.8332025-04-16evidence
o3-miniOpenAI0.7722025-01-30evidence
o4-miniOpenAI0.8142025-04-16evidence
Phi 4Microsoft0.5612024-12-12evidence
Phi 4 MiniMicrosoft0.2522025-02-01evidence
Phi 4 Mini ReasoningMicrosoft0.522025-04-30evidence
Phi 4 ReasoningMicrosoft0.6582025-04-30evidence
Phi 4 Reasoning PlusMicrosoft0.6892025-04-30evidence
Phi-3.5-mini-instructMicrosoft0.3042024-08-23evidence
Phi-3.5-MoE-instructMicrosoft0.3682024-08-23evidence
Qwen2 72B InstructQwen0.4242024-07-23evidence
Qwen2 7B InstructQwen0.2532024-07-23evidence
Qwen2.5 14B InstructQwen0.4552024-09-19evidence
Qwen2.5 32B InstructQwen0.4952024-09-19evidence
Qwen2.5 72B InstructQwen0.492024-09-19evidence
Qwen2.5 7B InstructQwen0.3642024-09-19evidence
Qwen2.5 VL 32B InstructQwen0.462025-02-28evidence
Qwen2.5-Omni-7BQwen0.3082025-03-27evidence
Qwen3 235B A22BQwen0.47472025-04-29evidence
Qwen3 30B A3BQwen0.6582025-04-29evidence
Qwen3 MaxQwen0.622025-12-15evidence
Qwen3 VL 30B A3B InstructQwen0.7042025-09-22evidence
Qwen3 VL 30B A3B ThinkingQwen0.7442025-09-22evidence
Qwen3 VL 32B InstructQwen0.6892025-09-22evidence
Qwen3 VL 32B ThinkingQwen0.7312025-09-22evidence
Qwen3 VL 4B ThinkingQwen0.6412025-09-22evidence
Qwen3 VL 8B ThinkingQwen0.6992025-09-22evidence
Qwen3-235B-A22B-Instruct-2507Qwen0.7752025-07-22evidence
Qwen3-235B-A22B-Thinking-2507Qwen0.8112025-07-25evidence
Qwen3-Next-80B-A3B-InstructQwen0.7292025-09-10evidence
Qwen3-Next-80B-A3B-ThinkingQwen0.7722025-09-10evidence
Qwen3.5-0.8BQwen0.1192026-03-02evidence
Qwen3.5-122B-A10BQwen0.8662026-02-24evidence
Qwen3.5-27BQwen0.8552026-02-24evidence
Qwen3.5-2BQwen0.5162026-03-02evidence
Qwen3.5-35B-A3BQwen0.8422026-02-24evidence
Qwen3.5-397B-A17BQwen0.8842026-02-16evidence
Qwen3.5-4BQwen0.7622026-03-02evidence
Qwen3.5-9BQwen0.8172026-03-02evidence
Qwen3.6 PlusQwen0.9042026-04-02evidence
Qwen3.6-27BQwen0.8782026-04-21evidence
Qwen3.6-35B-A3BQwen0.862026-04-16evidence
Qwen3.7 MaxQwen0.9242026-05-19evidence
Qwen3.7-PlusQwen0.9032026-05-31evidence
Qwen3.8 MaxQwen0.9262026-08-02evidence
Qwen3.8-27BQwen0.8922026-08-14evidence
QwQ-32BQwen0.6522025-03-05evidence
QwQ-32B-PreviewQwen0.6522024-11-28evidence
Sarvam-105BSarvam AI0.7872026-03-06evidence
Sarvam-30BSarvam AI0.6652026-03-06evidence
Seed 2.0 LiteByteDance0.8512026-02-14evidence
Seed 2.0 ProByteDance0.8892026-02-14evidence
Solar Pro 4Upstage0.892026-08-06evidence

Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.