Daily AI benchmark brief: 2026-10-18
The new Cost Per Correct benchmark ranks large language model agents by dollars spent per correct answer, not accuracy alone. Its five easy tasks use…
What changed in AI evaluation, and why it matters
Daily AI benchmark brief: 2026-10-17
NVIDIA’s garak v0.17.1 is an update to the existing security-evaluation tool. It adds hit@k, which reports whether an attack succeeded within the top k…
Daily AI benchmark brief: 2026-10-16
The new DIB-Bench release tests a gap that behavioral scoring cannot see: whether the deployed runtime has been tampered with while preserving expected…
Daily AI benchmark brief: 2026-10-15
A new audit found that all 761 traceable tumor test images in a widely used brain magnetic resonance imaging benchmark came from patients also represented…
Daily AI benchmark brief: 2026-10-14
Autonomous Science Bench is a new benchmark in today’s captured feed that makes an agent choose batches of experiments under a fixed budget, receive…
Daily AI benchmark brief: 2026-10-13
Unreachable Oracles introduces an offline audit for checking whether each benchmark answer can actually be derived from the information an agent receives…
Daily AI benchmark brief: 2026-10-12
MultiAgentInject is a new security benchmark with 4,050 trials that independently varies task, agent-network topology, attack class, stealth, and…
Daily AI benchmark brief: 2026-10-11
The new Two-Stage Measurement of Update Adherence benchmark tests whether a memory-enabled language-model agent follows a revised requirement rather than…
Daily AI benchmark brief: 2026-10-10
The new milipoint-runs artifact exposes split leakage in the MiliPoint radar benchmark: overlapping windows let most test examples share frames with…
Daily AI benchmark brief: 2026-10-09
TRACE is a new diagnostic protocol for checking whether an agent’s benchmark score changed because its behavior changed or because the evaluator changed…
Daily AI benchmark brief: 2026-10-08
A new runtime-verification study exposes a data-design gap in existing agent-safety benchmarks. It replayed AgentDojo and STAC trajectories through formal…
Daily AI benchmark brief: 2026-10-07
The new Agent Network Traffic benchmark (ANT) tests whether security teams can audit large language model agents through network traffic without…
Daily AI benchmark brief: 2026-10-05
The new “Passing the Test You Trained On” study evaluates 15 prompt-injection detectors on benign and injected outputs reconstructed from real agent…
Daily AI benchmark brief: 2026-10-04
The new 2026 Cryo-EM Benchmark Dataset provides 12 protein structures deposited after every test set disclosed for MICA, an artificial intelligence system…
Daily AI benchmark brief: 2026-10-03
The updated retinal benchmark audit provides image-level manifests for patient-grouped, source-matched, and leave-one-source-out evaluations, plus…
Daily AI benchmark brief: 2026-10-02
Video-Index is a new meta-benchmark that audits whether video tests actually require video understanding. Its five-level “attack pyramid” tests…
Daily AI benchmark brief: 2026-10-01
The new study “Dating the Model” reports that hidden current-date text injected into system prompts changed measured performance across nine large…
Daily AI benchmark brief: 2026-09-30
The newly released UserProxyBench tests a hidden dependency in interactive agent benchmarks: whether the language model playing the user follows its…
Daily AI benchmark brief: 2026-09-29
TraceDance is a new release that turns real agent deployment traces into targeted benchmarks for user-specified undesirable behaviors. It evaluates the…
Daily AI benchmark brief: 2026-09-28
The new Finance Agents Benchmark packages financial due diligence as an end-to-end agent test: 50 tasks, 160 documents, 231 grading criteria, an execution…
Daily AI benchmark brief: 2026-09-27
The new Kalyvox Voice Benchmark 2026 evaluates one voice agent through 240 controlled French and English calls across 12 scenario families. It reports…
Daily AI benchmark brief: 2026-09-26
The new repo2bench release converts Python repositories into coding-agent tasks mined from version history or created by removing functions. It admits…
Daily AI benchmark brief: 2026-09-25
The new PartHackBench release tests whether partial-credit scorers for long-running tool agents reward misleading progress. It compares honest and…
Daily AI benchmark brief: 2026-09-24
The new Uncheatable Eval benchmark evaluates base language models by measuring how efficiently they losslessly compress newly published text that is…
Daily AI benchmark brief: 2026-09-23
A new B3DB audit tests whether stereoisomers inflate blood–brain barrier prediction scores because common two-dimensional molecular representations map…
Daily AI benchmark brief: 2026-09-22
The new Curation-Bench release evaluates whether coding agents can improve training-data selection through an iterative workflow: agents inspect data…
Daily AI benchmark brief: 2026-09-20
The newly released TC-Slang Guardrail Benchmark evaluates Traditional Chinese prompt-injection defenses by slang year and pairs each slang example with a…
Daily AI benchmark brief: 2026-09-18
The new SonoCorpus and SonoBase release pairs 456,963 ultrasound images and 1,626,085 expert segmentation masks with an interactive segmentation model…
Daily AI benchmark brief: 2026-09-17
The new lexEN benchmark replaces disputed word-sense labels with a conservative, human-adjudicated correction layer: 211 labels changed and 56 removed…
Daily AI benchmark brief: 2026-09-16
A new audit of 254 SWE-bench coding-agent submissions reports that the leaderboard cannot reliably order its leading entries: the top two resolve the same…