HOLMES Benchmark

What is HOLMES?

HOLMES evaluates higher-order symbolic reasoning in language models using 1,379 natural-language problems paired with formalizations, answers, and verifiable…

Released

2026-06-22

Evaluates

Finance, Language & Knowledge, Reasoning

Openness

unknown

Importer

Claire Radar

Review status

ai-reviewed

Reported scores

0

Source provenance

• Original evidence https://arxiv.org/abs/2606.23238

HOLMES paper and code

• Paper https://arxiv.org/abs/2606.23238
• Code https://github.com/wuyucheng2002/HOLMES

Which sources cite HOLMES?

• HOLMES

Related Finance, Language & Knowledge, Reasoning benchmarks

• 5-Dialects-BN
• A Large-Scale Dataset and Benchmark
• A-OKVQA
• AARRI-Bench
• ABC-Bench
• ActiveFly-Bench
• ActiveSciBench
• AdaMem-Bench