Benchmark
AutomationBench-AA
Agentic SaaS workflows
- Categories
- agentic, tool-use, business
- Openness
- unknown
- Source
- artificial_analysis
- Reported scores
- 39
Reported scores
artificial_analysis
| Model | Organization | Reported value | Reported | Evidence |
|---|---|---|---|---|
| Claude 4.5 Haiku (Reasoning) | Anthropic | 0.09553946275772039 | 2025-10-15 | evidence |
| Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback) | Anthropic | 0.48581851262754977 | 2026-06-09 | evidence |
| Claude Opus 4.8 (Adaptive Reasoning, Max Effort) | Anthropic | 0.48529386173175704 | 2026-05-28 | evidence |
| Claude Sonnet 4.6 (Adaptive Reasoning, Max Effort) | Anthropic | 0.23823599456385894 | 2026-02-17 | evidence |
| Claude Sonnet 5 (Adaptive Reasoning, Max Effort) | Anthropic | 0.39190097119704015 | 2026-06-30 | evidence |
| DeepSeek V4 Flash (Reasoning, Max Effort) | DeepSeek | 0.16582172712070506 | 2026-04-24 | evidence |
| DeepSeek V4 Pro (Reasoning, Max Effort) | DeepSeek | 0.18827715939133852 | 2026-04-24 | evidence |
| Gemini 3.1 Flash-Lite | 0.10641533320288254 | 2026-03-03 | evidence | |
| Gemini 3.1 Pro Preview | 0.37535846144401597 | 2026-02-19 | evidence | |
| Gemini 3.5 Flash (high) | 0.42637546323234665 | 2026-05-19 | evidence | |
| Gemini 3.5 Flash-Lite | 0.3265910091801892 | 2026-07-21 | evidence | |
| Gemini 3.6 Flash (high) | 0.5107699915638565 | 2026-07-21 | evidence | |
| Gemini 3.7 Flash (high) | 0.6274556199710792 | 2026-08-13 | evidence | |
| GLM-5.2 (max) | Z.ai | 0.27844625079401897 | 2026-06-16 | evidence |
| GPT-5.5 (high) | OpenAI | 0.4016408791094086 | 2026-04-23 | evidence |
| GPT-5.5 (medium) | OpenAI | 0.37809962422121907 | 2026-04-23 | evidence |
| GPT-5.5 (xhigh) | OpenAI | 0.42103132391668596 | 2026-04-23 | evidence |
| GPT-5.5 Instant (June 2026) | OpenAI | 0.013565301670673362 | 2026-06-25 | evidence |
| GPT-5.6 Luna (max) | OpenAI | 0.4224225837450071 | 2026-07-09 | evidence |
| GPT-5.6 Sol (max) | OpenAI | 0.5118769513495001 | 2026-07-09 | evidence |
| GPT-5.6 Terra (max) | OpenAI | 0.4561448924929106 | 2026-07-09 | evidence |
| Grok 4.3 (high) | xAI | 0.08118872318781814 | 2026-04-30 | evidence |
| Grok 4.5 (high) | xAI | 0.5143740721186943 | 2026-07-08 | evidence |
| Inkling Small | Thinking Machines | 0.09395052631730252 | 2026-07-30 | evidence |
| KAT-Coder-Pro V1 | KwaiKAT | 0.09582156732403314 | 2025-11-11 | evidence |
| Kimi K2.6 | Moonshot AI | 0.19552711052930297 | 2026-04-20 | evidence |
| Kimi K2.7 Code | Moonshot AI | 0.22528586634678724 | 2026-06-12 | evidence |
| Kimi K3 (max) | Moonshot AI | 0.5270588443066796 | 2026-07-16 | evidence |
| MiMo-V2.5-Pro | Xiaomi | 0.1686244634925501 | 2026-04-22 | evidence |
| MiniMax-M3 | MiniMax | 0.1589777434670109 | 2026-06-01 | evidence |
| Mistral Medium 3.5 | Mistral | 0.13655123642959935 | 2026-04-29 | evidence |
| Nemotron 3 Ultra 550B A55B (Reasoning) | NVIDIA | 0.05670562350908316 | 2026-06-04 | evidence |
| Nex-N2-Pro | Nex AGI | 0.194033175065174 | 2026-06-02 | evidence |
| Qwen3.5 397B A17B (Reasoning) | Qwen | 0.12195605269463661 | 2026-02-16 | evidence |
| Qwen3.6 Plus | Qwen | 0.18645616010518853 | 2026-04-02 | evidence |
| Qwen3.7 Max | Qwen | 0.2557589370521027 | 2026-05-19 | evidence |
| Qwen3.7 Plus | Qwen | 0.20433540292579966 | 2026-06-01 | evidence |
| Ring-2.6-1T | InclusionAI | 0.021013617465460616 | 2026-05-08 | evidence |
| Step 3.7 Flash | StepFun | 0.0700460712463557 | 2026-05-29 | evidence |
Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.