SPIEval
Explore score history ↗SPIEval is a human-curated benchmark for evaluating large language models as mobile assistants that retrieve and reason over personal information scattered across multiple apps. It comprises 250 tasks across five cognitive capabilities, 4,335 fictional personal records in 10 simulated apps, and supports multi-turn interaction through 21 tools. Models receive underspecified user instructions and must search records and invoke tools; final execution calls are compared to human-annotated gold calls at the parameter level.
- Released
- 2026-08-11
- Institutions
- Tencent (China) · Fudan University
- Evaluates
- General AI, Language & Knowledge, Reasoning
- Openness
- unknown
- Importer
- Claire Radar
- Review status
- unreviewed
SPIEval paper and dataset
No reported scores are on record for this benchmark yet.