Benchmark
ERQA Benchmark
What is ERQA?
Embodied Reasoning Question Answering benchmark consisting of 400 multiple-choice visual questions across spatial reasoning, trajectory reasoning, action…
- Modality
- multimodal
- Evaluates
- reasoning, spatial reasoning, vision
- Openness
- unknown
- Importer
- LLM Stats
- Reported scores
- 24
Source provenance
- Original evidence https://api.zeroeval.com/leaderboard/benchmarks/erqa?top_n=500
ERQA code
ERQA results and reported scores
LLM Stats
| Model | Organization | Reported value | Reported | Evidence |
|---|---|---|---|---|
| GPT-4o | OpenAI | 0.352 | 2024-08-06 | evidence |
| GPT-5 | OpenAI | 0.657 | 2025-08-07 | evidence |
| Muse Spark | Meta | 0.647 | 2026-04-08 | evidence |
| o3 | OpenAI | 0.64 | 2025-04-16 | evidence |
| Qwen3 VL 235B A22B Instruct | Qwen | 0.513 | 2025-09-22 | evidence |
| Qwen3 VL 235B A22B Thinking | Qwen | 0.525 | 2025-09-22 | evidence |
| Qwen3 VL 30B A3B Instruct | Qwen | 0.43 | 2025-09-22 | evidence |
| Qwen3 VL 30B A3B Thinking | Qwen | 0.453 | 2025-09-22 | evidence |
| Qwen3 VL 32B Instruct | Qwen | 0.488 | 2025-09-22 | evidence |
| Qwen3 VL 32B Thinking | Qwen | 0.523 | 2025-09-22 | evidence |
| Qwen3 VL 4B Instruct | Qwen | 0.413 | 2025-09-22 | evidence |
| Qwen3 VL 4B Thinking | Qwen | 0.473 | 2025-09-22 | evidence |
| Qwen3 VL 8B Instruct | Qwen | 0.458 | 2025-09-22 | evidence |
| Qwen3 VL 8B Thinking | Qwen | 0.468 | 2025-09-22 | evidence |
| Qwen3.5-122B-A10B | Qwen | 0.62 | 2026-02-24 | evidence |
| Qwen3.5-27B | Qwen | 0.605 | 2026-02-24 | evidence |
| Qwen3.5-35B-A3B | Qwen | 0.648 | 2026-02-24 | evidence |
| Qwen3.6 Plus | Qwen | 0.657 | 2026-04-02 | evidence |
| Qwen3.6-27B | Qwen | 0.625 | 2026-04-21 | evidence |
| Qwen3.7-Plus | Qwen | 0.698 | 2026-05-31 | evidence |
| Qwen3.8 Max | Qwen | 0.778 | 2026-08-02 | evidence |
| Qwen3.8-27B | Qwen | 0.655 | 2026-08-14 | evidence |
| Seed 2.1 Pro | ByteDance | 0.72 | 2026-06-24 | evidence |
| Seed 2.1 Turbo | ByteDance | 0.713 | 2026-06-24 | evidence |
Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.