Benchmark Radar™
Star —

SPIEval

Explore score history ↗

SPIEval is a human-curated benchmark for evaluating large language models as mobile assistants that retrieve and reason over personal information scattered across multiple apps. It comprises 250 tasks across five cognitive capabilities, 4,335 fictional personal records in 10 simulated apps, and supports multi-turn interaction through 21 tools. Models receive underspecified user instructions and must search records and invoke tools; final execution calls are compared to human-annotated gold calls at the parameter level.

Released
2026-08-11
Institutions
Tencent (China) · Fudan University
Evaluates
General AI, Language & Knowledge, Reasoning
Openness
unknown
Importer
Claire Radar
Review status
unreviewed

SPIEval paper and dataset

No reported scores are on record for this benchmark yet.

Which sources cite SPIEval?

Related General AI, Language & Knowledge, Reasoning benchmarks