Benchmark

GPQA Benchmark

What is GPQA?

A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely…

Released
2023-11-20
Modality
text
Evaluates
physics, reasoning, general, biology, chemistry
Openness
restricted
Importer
LLM Stats
Reported scores
239

Source provenance

GPQA paper, code and dataset

GPQA results and reported scores

LLM Stats

ModelOrganizationReported valueReportedEvidence
ChatGPT-4o LatestOpenAI0.842024-05-13evidence
Claude 3 HaikuAnthropic0.3332024-03-13evidence
Claude 3 OpusAnthropic0.5042024-02-29evidence
Claude 3 SonnetAnthropic0.4042024-02-29evidence
Claude 3.5 HaikuAnthropic0.4162024-10-22evidence
Claude 3.5 SonnetAnthropic0.5942024-06-21evidence
Claude 3.5 SonnetAnthropic0.6722024-10-22evidence
Claude 3.7 SonnetAnthropic0.8482025-02-24evidence
Claude Haiku 4.5Anthropic0.732025-10-15evidence
Claude Mythos PreviewAnthropic0.9462026-04-07evidence
Claude Opus 4Anthropic0.7962025-05-22evidence
Claude Opus 4.1Anthropic0.8092025-08-05evidence
Claude Opus 4.5Anthropic0.872025-11-24evidence
Claude Opus 4.6Anthropic0.9132026-02-05evidence
Claude Opus 4.7Anthropic0.9422026-04-16evidence
Claude Opus 4.8Anthropic0.9362026-05-28evidence
Claude Sonnet 4Anthropic0.7542025-05-22evidence
Claude Sonnet 4.5Anthropic0.8342025-09-29evidence
Claude Sonnet 4.6Anthropic0.8992026-02-17evidence
DeepSeek R1 Distill Llama 70BDeepSeek0.6522025-01-20evidence
DeepSeek R1 Distill Llama 8BDeepSeek0.492025-01-20evidence
DeepSeek R1 Distill Qwen 1.5BDeepSeek0.3382025-01-20evidence
DeepSeek R1 Distill Qwen 14BDeepSeek0.5912025-01-20evidence
DeepSeek R1 Distill Qwen 32BDeepSeek0.6212025-01-20evidence
DeepSeek R1 Distill Qwen 7BDeepSeek0.4912025-01-20evidence
DeepSeek R1 ZeroDeepSeek0.7332025-01-20evidence
DeepSeek-R1-0528DeepSeek0.812025-05-28evidence
DeepSeek-V3DeepSeek0.5912024-12-25evidence
DeepSeek-V3 0324DeepSeek0.6842025-03-25evidence
DeepSeek-V3.1DeepSeek0.7492025-01-10evidence
DeepSeek-V3.2DeepSeek0.8242025-12-01evidence
DeepSeek-V3.2 (Thinking)DeepSeek0.8242025-12-01evidence
DeepSeek-V3.2-ExpDeepSeek0.7992025-09-29evidence
DeepSeek-V4-Flash-0423DeepSeek0.8742026-04-23evidence
DeepSeek-V4-Flash-MaxDeepSeek0.8812026-04-23evidence
DeepSeek-V4-Pro-MaxDeepSeek0.9012026-04-23evidence
DiffusionGemma 26B-A4BGoogle0.7322026-06-10evidence
ERNIE 4.5Baidu0.742025-06-25evidence
ERNIE 5.0Baidu0.852026-01-22evidence
Gemini 1.0 ProGoogle0.2792024-02-15evidence
Gemini 1.5 FlashGoogle0.512024-05-01evidence
Gemini 1.5 Flash 8BGoogle0.3842024-03-15evidence
Gemini 1.5 ProGoogle0.5912024-05-01evidence
Gemini 2.0 FlashGoogle0.6212024-12-01evidence
Gemini 2.0 Flash ThinkingGoogle0.7422025-01-21evidence
Gemini 2.0 Flash-LiteGoogle0.5152025-02-05evidence
Gemini 2.5 FlashGoogle0.8282025-05-20evidence
Gemini 2.5 Flash-LiteGoogle0.6462025-06-17evidence
Gemini 2.5 ProGoogle0.832025-05-20evidence
Gemini 2.5 Pro Preview 06-05Google0.8642025-06-05evidence
Gemini 3 FlashGoogle0.9042025-12-17evidence
Gemini 3 ProGoogle0.9192025-11-18evidence
Gemini 3.1 Flash-LiteGoogle0.8692026-03-03evidence
Gemini 3.1 ProGoogle0.9432026-02-19evidence
Gemini DiffusionGoogle0.4042025-05-20evidence
Gemma 3 12BGoogle0.4092025-03-12evidence
Gemma 3 1BGoogle0.1922025-03-12evidence
Gemma 3 27BGoogle0.4242025-03-12evidence
Gemma 3 4BGoogle0.3082025-03-12evidence
Gemma 3n E2B InstructedGoogle0.2482025-06-26evidence
Gemma 3n E2B Instructed LiteRT (Preview)Google0.2482025-05-20evidence
Gemma 3n E4B InstructedGoogle0.2372025-06-26evidence
Gemma 3n E4B Instructed LiteRT PreviewGoogle0.2372025-05-20evidence
Gemma 4 12BGoogle0.7882026-05-23evidence
Gemma 4 26B-A4BGoogle0.8232026-04-02evidence
Gemma 4 31BGoogle0.8432026-04-02evidence
Gemma 4 E2BGoogle0.4342026-04-02evidence
Gemma 4 E4BGoogle0.5862026-04-02evidence
GLM-4.5Z.ai0.7912025-07-28evidence
GLM-4.5-AirZ.ai0.752025-07-28evidence
GLM-4.6Z.ai0.812025-09-30evidence
GLM-4.7Z.ai0.8572025-12-22evidence
GLM-4.7-FlashZ.ai0.7522026-01-19evidence
GLM-5.1Z.ai0.8622026-04-07evidence
GLM-5.2Z.ai0.9122026-06-16evidence
GPT OSS 120BOpenAI0.8012025-08-05evidence
GPT OSS 120B HighOpenAI0.8092025-08-05evidence
GPT OSS 20BOpenAI0.7152025-08-05evidence
GPT OSS 20B HighOpenAI0.7422025-08-05evidence
GPT-3.5 TurboOpenAI0.3082023-03-21evidence
GPT-4OpenAI0.3572023-06-13evidence
GPT-4 TurboOpenAI0.482024-04-09evidence
GPT-4.1OpenAI0.6632025-04-14evidence
GPT-4.1 miniOpenAI0.652025-04-14evidence
GPT-4.1 nanoOpenAI0.5032025-04-14evidence
GPT-4.5OpenAI0.6952025-02-27evidence
GPT-4oOpenAI0.5362024-05-13evidence
GPT-4oOpenAI0.7012024-08-06evidence
GPT-4o miniOpenAI0.4022024-07-18evidence
GPT-5OpenAI0.8572025-08-07evidence
GPT-5 HighOpenAI0.8732025-08-07evidence
GPT-5 MediumOpenAI0.8812025-08-07evidence
GPT-5 miniOpenAI0.8232025-08-07evidence
GPT-5 nanoOpenAI0.7122025-08-07evidence
GPT-5.1OpenAI0.8812025-11-13evidence
GPT-5.1 HighOpenAI0.8812025-11-12evidence
GPT-5.1 InstantOpenAI0.8812025-11-12evidence
GPT-5.1 ThinkingOpenAI0.8812025-11-12evidence
GPT-5.2OpenAI0.9242025-12-11evidence
GPT-5.2 ProOpenAI0.9322025-12-11evidence
GPT-5.4OpenAI0.9282026-03-05evidence
GPT-5.4 miniOpenAI0.882026-03-17evidence
GPT-5.4 nanoOpenAI0.8282026-03-17evidence
GPT-5.5OpenAI0.9362026-04-23evidence
GPT-5.5 InstantOpenAI0.8562026-05-05evidence
GPT-5.6 LunaOpenAI0.9232026-07-09evidence
GPT-5.6 SolOpenAI0.9462026-07-09evidence
GPT-5.6 TerraOpenAI0.9292026-07-09evidence
Grok 4 FastxAI0.8572025-08-28evidence
Grok 4.5xAI0.932026-07-16evidence
Grok-1.5xAI0.3592024-03-28evidence
Grok-2xAI0.562024-08-13evidence
Grok-2 minixAI0.512024-08-13evidence
Grok-3xAI0.8462025-02-17evidence
Grok-3 MinixAI0.842025-02-17evidence
Grok-4xAI0.8752025-07-09evidence
Grok-4 HeavyxAI0.8842025-07-09evidence
Hermes 3 70BNous Research0.6612024-08-15evidence
Hy3Tencent0.9042026-07-06evidence
Inkling-SmallThinking Machines Lab0.8952026-07-30evidence
Jamba 1.5 LargeAI21 Labs0.3692024-08-22evidence
Jamba 1.5 MiniAI21 Labs0.3232024-08-22evidence
Kimi K2 0905Moonshot AI0.7582025-09-05evidence
Kimi K2 BaseMoonshot AI0.4812025-07-11evidence
Kimi K2 InstructMoonshot AI0.7512025-07-11evidence
Kimi K2-Instruct-0905Moonshot AI0.7512025-09-05evidence
Kimi K2-Thinking-0905Moonshot AI0.8452025-09-05evidence
Kimi K2.5Moonshot AI0.8762026-01-27evidence
Kimi K2.6Moonshot AI0.9052026-04-20evidence
Kimi K3Moonshot AI0.9352026-07-16evidence
Llama 3.1 405B InstructMeta0.5072024-07-23evidence
Llama 3.1 70B InstructMeta0.4172024-07-23evidence
Llama 3.1 8B InstructMeta0.3042024-07-23evidence
Llama 3.1 Nemotron Nano 8B V1NVIDIA0.5412025-03-18evidence
Llama 3.1 Nemotron Ultra 253B v1NVIDIA0.76012025-04-07evidence
Llama 3.2 11B InstructMeta0.3282024-09-25evidence
Llama 3.2 3B InstructMeta0.3282024-09-25evidence
Llama 3.2 90B InstructMeta0.4672024-09-25evidence
Llama 3.3 70B InstructMeta0.5052024-12-06evidence
Llama 4 MaverickMeta0.6982025-04-05evidence
Llama 4 ScoutMeta0.5722025-04-05evidence
Llama-3.3 Nemotron Super 49B v1NVIDIA0.66672025-03-18evidence
LongCat-Flash-ChatMeituan0.73232025-08-29evidence
LongCat-Flash-LiteMeituan0.66782026-02-05evidence
LongCat-Flash-ThinkingMeituan0.8152025-09-22evidence
LongCat-Flash-Thinking-2601Meituan0.8052026-01-14evidence
Magistral MediumMistral0.7082025-06-10evidence
Magistral Small 2506Mistral0.68182025-06-10evidence
MAI-Code-1-FlashMicrosoft0.8462026-06-02evidence
MAI-Thinking-1Microsoft0.8422026-06-02evidence
Mercury 2Inception0.742026-02-24evidence
MiMo-V2-FlashXiaomi0.8372025-12-16evidence
MiMo-V2.5-ProXiaomi0.6672026-04-27evidence
Min istral 3 (3B Reasoning 2512)Mistral0.5342025-12-04evidence
MiniMax M1 40KMiniMax0.6922025-06-16evidence
MiniMax M1 80KMiniMax0.72025-06-16evidence
MiniMax M2MiniMax0.782025-10-27evidence
MiniMax M2.1MiniMax0.812025-12-23evidence
Ministral 3 (14B Reasoning 2512)Mistral0.7122025-12-04evidence
Ministral 3 (8B Reasoning 2512)Mistral0.6682025-12-04evidence
Mistral Large 3 (675B Base)Mistral0.4392025-12-04evidence
Mistral Large 3 (675B Instruct 2512 Eagle)Mistral0.4392025-12-04evidence
Mistral Large 3 (675B Instruct 2512 NVFP4)Mistral0.4392025-12-04evidence
Mistral Large 3 (675B Instruct 2512)Mistral0.4392025-12-04evidence
Mistral Small 3 24B BaseMistral0.34372025-01-30evidence
Mistral Small 3 24B InstructMistral0.4532025-01-30evidence
Mistral Small 3.1 24B BaseMistral0.3752025-03-17evidence
Mistral Small 3.1 24B InstructMistral0.45962025-03-17evidence
Mistral Small 3.2 24B InstructMistral0.46132025-06-20evidence
Mistral Small 4Mistral0.7122026-03-16evidence
Muse Glimmer-30BMeta0.8352026-08-10evidence
Muse SparkMeta0.8952026-04-08evidence
Nemotron 3 Nano (30B A3B)NVIDIA0.752025-12-15evidence
Nemotron 3 Super (120B A12B)NVIDIA0.8272026-03-11evidence
Nemotron 3 Ultra (550B A55B)NVIDIA0.872026-06-04evidence
Nemotron 3.5 Lightning (30B A3B)NVIDIA0.75442026-08-11evidence
Nemotron Nano 9B v2NVIDIA0.642025-08-18evidence
Nova 2 LiteAmazon0.7962025-12-02evidence
Nova 2 ProAmazon0.8142025-12-02evidence
Nova LiteAmazon0.422024-11-20evidence
Nova MicroAmazon0.42024-11-20evidence
Nova ProAmazon0.4692024-11-20evidence
o1OpenAI0.782024-12-17evidence
o1-miniOpenAI0.62024-09-12evidence
o1-previewOpenAI0.7332024-09-12evidence
o1-proOpenAI0.792024-12-17evidence
o3OpenAI0.8332025-04-16evidence
o3-miniOpenAI0.7722025-01-30evidence
o4-miniOpenAI0.8142025-04-16evidence
Phi 4Microsoft0.5612024-12-12evidence
Phi 4 MiniMicrosoft0.2522025-02-01evidence
Phi 4 Mini ReasoningMicrosoft0.522025-04-30evidence
Phi 4 ReasoningMicrosoft0.6582025-04-30evidence
Phi 4 Reasoning PlusMicrosoft0.6892025-04-30evidence
Phi-3.5-mini-instructMicrosoft0.3042024-08-23evidence
Phi-3.5-MoE-instructMicrosoft0.3682024-08-23evidence
Qwen2 72B InstructQwen0.4242024-07-23evidence
Qwen2 7B InstructQwen0.2532024-07-23evidence
Qwen2.5 14B InstructQwen0.4552024-09-19evidence
Qwen2.5 32B InstructQwen0.4952024-09-19evidence
Qwen2.5 72B InstructQwen0.492024-09-19evidence
Qwen2.5 7B InstructQwen0.3642024-09-19evidence
Qwen2.5 VL 32B InstructQwen0.462025-02-28evidence
Qwen2.5-Omni-7BQwen0.3082025-03-27evidence
Qwen3 235B A22BQwen0.47472025-04-29evidence
Qwen3 30B A3BQwen0.6582025-04-29evidence
Qwen3 MaxQwen0.622025-12-15evidence
Qwen3 VL 30B A3B InstructQwen0.7042025-09-22evidence
Qwen3 VL 30B A3B ThinkingQwen0.7442025-09-22evidence
Qwen3 VL 32B InstructQwen0.6892025-09-22evidence
Qwen3 VL 32B ThinkingQwen0.7312025-09-22evidence
Qwen3 VL 4B ThinkingQwen0.6412025-09-22evidence
Qwen3 VL 8B ThinkingQwen0.6992025-09-22evidence
Qwen3-235B-A22B-Instruct-2507Qwen0.7752025-07-22evidence
Qwen3-235B-A22B-Thinking-2507Qwen0.8112025-07-25evidence
Qwen3-Next-80B-A3B-InstructQwen0.7292025-09-10evidence
Qwen3-Next-80B-A3B-ThinkingQwen0.7722025-09-10evidence
Qwen3.5-0.8BQwen0.1192026-03-02evidence
Qwen3.5-122B-A10BQwen0.8662026-02-24evidence
Qwen3.5-27BQwen0.8552026-02-24evidence
Qwen3.5-2BQwen0.5162026-03-02evidence
Qwen3.5-35B-A3BQwen0.8422026-02-24evidence
Qwen3.5-397B-A17BQwen0.8842026-02-16evidence
Qwen3.5-4BQwen0.7622026-03-02evidence
Qwen3.5-9BQwen0.8172026-03-02evidence
Qwen3.6 PlusQwen0.9042026-04-02evidence
Qwen3.6-27BQwen0.8782026-04-21evidence
Qwen3.6-35B-A3BQwen0.862026-04-16evidence
Qwen3.7 MaxQwen0.9242026-05-19evidence
Qwen3.7-PlusQwen0.9032026-05-31evidence
Qwen3.8 MaxQwen0.9262026-08-02evidence
Qwen3.8-27BQwen0.8922026-08-14evidence
QwQ-32BQwen0.6522025-03-05evidence
QwQ-32B-PreviewQwen0.6522024-11-28evidence
Sarvam-105BSarvam AI0.7872026-03-06evidence
Sarvam-30BSarvam AI0.6652026-03-06evidence
Seed 2.0 LiteByteDance0.8512026-02-14evidence
Seed 2.0 ProByteDance0.8892026-02-14evidence
Solar Pro 4Upstage0.892026-08-06evidence

Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.

Which sources cite GPQA?

Related physics, reasoning, general, biology, chemistry benchmarks