ORCA-bench
Explore score history ↗ORCA-bench evaluates language model agents on root cause analysis in a production-fidelity oncall setting, using a live OpenTelemetry-instrumented microservice system with 1,079 tasks varying in report specificity, time-to-detection, and fault co-occurrence. Agents access metrics, logs, traces, and source code via real telemetry interfaces.
- Released
- 2026-07-30
- Institutions
- Cornell Tech · Columbia University
- Evaluates
- Code & Software, Software & AI Compute, cs.CL
- Openness
- unknown
- Importer
- Claire Radar
- Review status
- unreviewed
ORCA-bench paper and code
No reported scores are on record for this benchmark yet.