Benchmark
HumanEval
A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing…
- Released
- 2021-07-08
- Modality
- text
- Categories
- reasoning, code
- Openness
- unknown
- Source
- llm_stats
- Reported scores
- 66
Reported scores
llm_stats
| Model | Organization | Reported value | Reported | Evidence |
|---|---|---|---|---|
| Claude 3 Haiku | Anthropic | 0.759 | 2024-03-13 | evidence |
| Claude 3 Opus | Anthropic | 0.849 | 2024-02-29 | evidence |
| Claude 3 Sonnet | Anthropic | 0.73 | 2024-02-29 | evidence |
| Claude 3.5 Haiku | Anthropic | 0.881 | 2024-10-22 | evidence |
| Claude 3.5 Sonnet | Anthropic | 0.92 | 2024-06-21 | evidence |
| Claude 3.5 Sonnet | Anthropic | 0.937 | 2024-10-22 | evidence |
| Codestral-22B | Mistral | 0.811 | 2024-05-29 | evidence |
| DeepSeek-V2.5 | DeepSeek | 0.89 | 2024-05-08 | evidence |
| Gemini 1.5 Flash | 0.743 | 2024-05-01 | evidence | |
| Gemini 1.5 Pro | 0.841 | 2024-05-01 | evidence | |
| Gemini Diffusion | 0.896 | 2025-05-20 | evidence | |
| Gemma 2 27B | 0.518 | 2024-06-27 | evidence | |
| Gemma 2 9B | 0.402 | 2024-06-27 | evidence | |
| Gemma 3 12B | 0.854 | 2025-03-12 | evidence | |
| Gemma 3 1B | 0.415 | 2025-03-12 | evidence | |
| Gemma 3 27B | 0.878 | 2025-03-12 | evidence | |
| Gemma 3 4B | 0.713 | 2025-03-12 | evidence | |
| Gemma 3n E2B Instructed | 0.665 | 2025-06-26 | evidence | |
| Gemma 3n E2B Instructed LiteRT (Preview) | 0.665 | 2025-05-20 | evidence | |
| Gemma 3n E4B Instructed | 0.75 | 2025-06-26 | evidence | |
| Gemma 3n E4B Instructed LiteRT Preview | 0.75 | 2025-05-20 | evidence | |
| GPT-3.5 Turbo | OpenAI | 0.68 | 2023-03-21 | evidence |
| GPT-4 | OpenAI | 0.67 | 2023-06-13 | evidence |
| GPT-4 Turbo | OpenAI | 0.871 | 2024-04-09 | evidence |
| GPT-4.5 | OpenAI | 0.88 | 2025-02-27 | evidence |
| GPT-4o | OpenAI | 0.902 | 2024-05-13 | evidence |
| GPT-4o mini | OpenAI | 0.872 | 2024-07-18 | evidence |
| GPT-5 | OpenAI | 0.934 | 2025-08-07 | evidence |
| Granite 3.3 8B Base | IBM | 0.8973 | 2025-04-16 | evidence |
| Granite 3.3 8B Instruct | IBM | 0.8973 | 2025-04-16 | evidence |
| Grok-1.5 | xAI | 0.741 | 2024-03-28 | evidence |
| Grok-2 | xAI | 0.884 | 2024-08-13 | evidence |
| Grok-2 mini | xAI | 0.857 | 2024-08-13 | evidence |
| IBM Granite 4.0 Tiny Preview | IBM | 0.824 | 2025-05-02 | evidence |
| Kimi K2 0905 | Moonshot AI | 0.945 | 2025-09-05 | evidence |
| Kimi K2 Instruct | Moonshot AI | 0.933 | 2025-07-11 | evidence |
| Llama 3.1 405B Instruct | Meta | 0.89 | 2024-07-23 | evidence |
| Llama 3.1 70B Instruct | Meta | 0.805 | 2024-07-23 | evidence |
| Llama 3.1 8B Instruct | Meta | 0.726 | 2024-07-23 | evidence |
| Llama 3.3 70B Instruct | Meta | 0.884 | 2024-12-06 | evidence |
| LongCat-Flash-Chat | Meituan | 0.8841 | 2025-08-29 | evidence |
| MiniCPM-SALA | OpenBMB | 0.9512 | 2026-02-11 | evidence |
| Ministral 8B Instruct | Mistral | 0.348 | 2024-10-16 | evidence |
| Mistral Large 2 | Mistral | 0.92 | 2024-07-24 | evidence |
| Mistral Small 3 24B Instruct | Mistral | 0.848 | 2025-01-30 | evidence |
| Mistral Small 3.1 24B Instruct | Mistral | 0.8841 | 2025-03-17 | evidence |
| Nova Lite | Amazon | 0.854 | 2024-11-20 | evidence |
| Nova Micro | Amazon | 0.811 | 2024-11-20 | evidence |
| Nova Pro | Amazon | 0.89 | 2024-11-20 | evidence |
| o1 | OpenAI | 0.881 | 2024-12-17 | evidence |
| o1-mini | OpenAI | 0.924 | 2024-09-12 | evidence |
| Phi 4 | Microsoft | 0.826 | 2024-12-12 | evidence |
| Phi-3.5-mini-instruct | Microsoft | 0.628 | 2024-08-23 | evidence |
| Phi-3.5-MoE-instruct | Microsoft | 0.707 | 2024-08-23 | evidence |
| Pixtral-12B | Mistral | 0.72 | 2024-09-17 | evidence |
| Qwen2 72B Instruct | Qwen | 0.86 | 2024-07-23 | evidence |
| Qwen2 7B Instruct | Qwen | 0.799 | 2024-07-23 | evidence |
| Qwen2.5 14B Instruct | Qwen | 0.835 | 2024-09-19 | evidence |
| Qwen2.5 32B Instruct | Qwen | 0.884 | 2024-09-19 | evidence |
| Qwen2.5 72B Instruct | Qwen | 0.866 | 2024-09-19 | evidence |
| Qwen2.5 7B Instruct | Qwen | 0.848 | 2024-09-19 | evidence |
| Qwen2.5 VL 32B Instruct | Qwen | 0.915 | 2025-02-28 | evidence |
| Qwen2.5-Coder 32B Instruct | Qwen | 0.927 | 2024-09-19 | evidence |
| Qwen2.5-Coder 7B Instruct | Qwen | 0.884 | 2024-09-19 | evidence |
| Qwen2.5-Omni-7B | Qwen | 0.787 | 2025-03-27 | evidence |
| Sarvam-30B | Sarvam AI | 0.921 | 2026-03-06 | evidence |
Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.