Benchmark
Bird-SQL (dev) Benchmark
What is Bird-SQL (dev)?
BIRD (BIg Bench for LaRge-scale Database Grounded Text-to-SQLs) is a comprehensive text-to-SQL benchmark containing 12,751 question-SQL pairs across 95…
- Released
- 2023-05-04
- Modality
- text
- Evaluates
- reasoning
- Openness
- unknown
- Importer
- LLM Stats
- Reported scores
- 7
Source provenance
Bird-SQL (dev) results and reported scores
LLM Stats
| Model | Organization | Reported value | Reported | Evidence |
|---|---|---|---|---|
| Gemini 2.0 Flash | 0.569 | 2024-12-01 | evidence | |
| Gemini 2.0 Flash-Lite | 0.574 | 2025-02-05 | evidence | |
| Gemma 3 12B | 0.479 | 2025-03-12 | evidence | |
| Gemma 3 1B | 0.064 | 2025-03-12 | evidence | |
| Gemma 3 27B | 0.544 | 2025-03-12 | evidence | |
| Gemma 3 4B | 0.363 | 2025-03-12 | evidence | |
| Nemotron 3 Super (120B A12B) | NVIDIA | 0.418 | 2026-03-11 | evidence |
Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.