Benchmark Radar™
Star —

Beyond Leaderboard Ranks and 5 more in AI benchmark news · Nov 17, 2026

635evidence observations
11sources represented
15public-attention signals

Featured on this day

  1. Beyond Leaderboard Ranks: Selecting Software Engineering Agents Under Budget Constraints
  2. EVIDENT: Evaluating Evidence Grounding in GUI Agents
  3. Benchmark Factory: Agentic Benchmark Creation via Dataset Curation and Label-Free Quality Anchors
  4. DetourBench: Trajectory-Level Evaluation of Interactive Agents with Exact Minimum-Action Bounds
  5. When Helpful Agents Cause Financial Harm: Evaluating LLM Agent Safety in Executable Economic Worlds
  6. Accelerate Bench: Evaluating Agents for Application-Level Program Optimization

Daily briefing

  1. The new study “Beyond Leaderboard Ranks” reframes software-agent selection as an accuracy, cost, and uncertainty decision. Across 58,197 agent runs on seven benchmarks, 27 of 35 adjacent options on the accuracy-cost front had statistically unresolved accuracy differences, meaning their apparent ordering did not reliably distinguish performance. [E006] Why it matters: If you are buying or evaluating coding agents under a budget, this means a leaderboard winner may not justify its higher operating cost. Selection reports need confidence intervals and the cost of running enough evaluations to resolve meaningful differences, rather than rank and average accuracy alone. Evidence: E006. High confidence.
  2. EVIDENT is a new benchmark that tests whether a web-browsing agent’s answer is supported by evidence it actually observed, rather than merely checking task completion. Its 220 English and Korean tasks include human-curated records of what evidence must be verified, and separate metrics score completion, evidence use, and their combination. [E012] Why it matters: If you are selecting a graphical user interface agent for research, compliance, or fact-finding, this design can reveal agents that reach plausible answers without inspecting the necessary page content. It changes the evaluation target from successful output alone to an auditable connection between navigation and answer. Evidence: E012. High confidence.
  3. Benchmark Factory is a new system for constructing coding-agent benchmarks from an organization’s own completed code changes. An automated agent writes sourcing queries, tests candidate tasks with models of known relative strength, diagnoses failures, and revises the benchmark until it meets quality checks that do not require a human label for every task; humans remain at checkpoints. [E007] Why it matters: If public coding benchmarks do not resemble your repositories, this offers a concrete alternative to fully manual internal benchmark creation. The decision becomes whether its automated quality checks and sampled human review provide enough assurance for model selection in your own development environment. Evidence: E007. Medium confidence.
  4. DetourBench, newly discovered in today’s feed, evaluates not only whether an interactive agent succeeds but how many actions it takes relative to an exactly computed minimum. It constrains each task to a finite environment where the shortest possible evidence-gathering route can be calculated. [E105] Why it matters: If agent actions carry latency, monetary cost, or exposure to sensitive data, endpoint success can hide wasteful exploration. This benchmark design lets evaluators distinguish direct evidence collection from broad, unnecessary browsing and identify agents that stop just short of the required evidence. Evidence: E105. Medium confidence.
  5. EconSafeBench is a new safety benchmark with 828 cases across five executable economic scenarios. It matches three sources of pressure—explicit user instructions, other actors, and the agent’s own performance objective—and includes interactions lasting 30 periods rather than only static prompts. [E013] Why it matters: If you are evaluating agents that make financial or resource-allocation decisions, this means safety testing can include harmful behavior that emerges while pursuing a seemingly legitimate objective over time. That is a different product decision signal from refusal tests based only on explicitly unsafe user requests. Evidence: E013. High confidence.
  6. Accelerate Bench is a new application-level optimization benchmark containing 32 central processing unit tasks adapted from scientific software and computing competitions. Each task specifies correctness requirements, timing boundaries, workload, and permitted edits; agents can iterate against public tests, while a separate verifier scores final submissions. [E010] Why it matters: If you are choosing a coding agent for performance engineering, this evaluates the full model-and-tool-wrapper combination on coordinated program changes, not isolated code snippets. The independent final verifier also reduces the risk that an agent improves measured speed by changing the required computation. Evidence: E010. High confidence.

What arrived

What benchmarks, datasets, or evaluation methods did the radar first see today?

high confidenceNot enough evidence

Notable releases include OpenMAS-GApp for multi-agent application construction, MLIRBench for compiler reasoning, ExtremeDrive for critical driving, CataTQA for scientific-table question answering, Accelerate Bench for program optimization, EVIDENT for evidence-grounded GUI agents, and EconSafeBench for economic-agent safety. The feed separately discovered ReSpaceBench, CAE-Bench, DetourBench, SWE-CoWork, HospitalBench, and Omni-Workspace. These are examples rather than a complete inventory.

The captured feed added evaluations spanning agent construction, software and compiler work, driving, scientific tables, web evidence, economic safety, spatial memory, engineering solvers, workplace coding, hospitals, and mixed-media office tasks. It also surfaced statistical evaluation methods for agent reliability and benchmark uncertainty. Each artifact’s event label matters: some were recorded as releases, while others were merely discovered by the radar.

Takeaway: Today’s first observations show broad coverage, with agent evaluations especially visible among the highlighted records. Treat this as a map of what entered this keyword-filtered feed, not as a representative account of the field or proof that every artifact itself originated today. A complete artifact-level inventory was not supplied in the evidence packet.

Another reading: First observation by the radar is not equivalent to a new release. For example, SHAPE benchmark data carried a publication time before the capture day even though the feed classified its record as released. The packet also covers only selected evidence, so less prominent first observations may materially broaden or alter this summary.

  • S022 artifacts first observed by the radar today: 310

Which of today's arrivals document how they score an answer?

high confidenceNot enough evidence

The clearest documented scoring arrivals are Accelerate Bench, which uses a separate final-submission verifier; VAHBench, which combines delivery, quality, reliability, and reporting checks; EVIDENT, which separates task completion from evidence-grounded success; SchedOps-Bench, which judges server end states; DetourBench, which compares trajectories with computable minimum-action bounds; SWE-CoWork, which uses private executable tests; and HospitalBench, which scores linked subtasks and diagnoses where a trajectory breaks.

Other concrete examples include the dettato benchmark, which publishes outputs and scoring scripts; the Gujarati speech leaderboard, which reports character and word error measures with uncertainty testing; and Ai_Evals, which combines expected-answer datasets, fixed checks, model-based judging, and pass-or-fail tests. These records explain more than a leaderboard result: they identify the checks, verifier, test procedure, or score components used to judge an answer.

Takeaway: For auditable evaluation work, prioritize arrivals that expose executable tests, scoring scripts, end-state checks, evidence requirements, or explicit trajectory bounds. The registry does not contain a computed count of all first-observed artifacts with documented scoring, and the selected packet is insufficient for an exhaustive list.

Another reading: Naming score dimensions is not the same as publishing a reproducible scoring rule. For example, VAHBench describes completeness, quality, reliability, and reporting checks, but the supplied summary does not expose their formulas or aggregation. Full papers, repositories, and evaluator code would be needed to determine whether every listed method can be independently reproduced.

  • S022 artifacts first observed by the radar today: 310

What is still moving

Which artifacts the radar already tracked moved measurably, and over what span?

medium confidenceNot enough evidence

The registry flags download movement for lmarena-ai/leaderboard-dataset, the privacy-utility evaluation results, the Agnuxo benchmark, open-llm-leaderboard/requests, web-trajectories-preference-raw, explicit-edit-benchmark, GEMS-GER, and the Danish ASR leaderboard. Each movement is cumulative across its full tracked span, not a one-day change.

The spans run respectively from late July, late October, late July, early September, early November, mid-September, yesterday, and mid-August through today. Downloads increased for most listed artifacts but decreased for open-llm-leaderboard/requests and GEMS-GER.

Takeaway: These are counter movements within this keyword-filtered captured feed, not evidence of releases, updates, quality changes, or field-wide adoption. Artifact-level evidence records with E-identifiers were not supplied, so the registry results cannot be independently documented here.

Another reading: Download counters can be revised or reset, so even large cumulative changes may not represent actual new usage. The negative movements make that measurement limitation especially relevant.

  • S025 downloads change for lmarena-ai/leaderboard-dataset: 315,458.0 downloads
  • S026 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 39,722.0 downloads
  • S027 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 31,001.0 downloads
  • S028 downloads change for open-llm-leaderboard/requests: -28,441.0 downloads
  • S029 downloads change for lukakuma/web-trajectories-preference-raw: 23,620.0 downloads
  • S030 downloads change for alexshpunt/explicit-edit-benchmark: 13,800.0 downloads
  • S031 downloads change for GEMS-GER: A Machine Learning Benchmark Dataset of Long-Term Groundwater Levels in Germany with Meteorological Forcings and Site-Specific Environmental Features: -9,555.0 downloads
  • S032 downloads change for RyeAI/danish-asr-leaderboard: 5,367.0 downloads

Which of that movement is corroborated by more than one data source?

high confidenceNot enough evidence

None of the registry-listed download movements is corroborated by more than one source. Each was measured by a single source over its stated full tracked span.

Seeing an artifact in multiple places is not enough. More than one source must report the same metric movement. The registry does not provide such confirmation for any listed download change.

Takeaway: Treat all listed movement as single-source measurement within this captured feed. The registry does not support a corroborated-movement claim, and no artifact-level E-evidence was supplied.

Another reading: The feed did see some tracked artifacts through multiple sources, but that statistic explicitly does not mean those sources measured the same metric. Separate tracked-artifact metadata also marks some entries corroborated, yet no matching movement statistic or E-evidence is available to validate them.

  • S024 tracked artifacts today seen by more than one data source: 13
  • S025 downloads change for lmarena-ai/leaderboard-dataset: 315,458.0 downloads
  • S026 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 39,722.0 downloads
  • S027 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 31,001.0 downloads
  • S028 downloads change for open-llm-leaderboard/requests: -28,441.0 downloads
  • S029 downloads change for lukakuma/web-trajectories-preference-raw: 23,620.0 downloads
  • S030 downloads change for alexshpunt/explicit-edit-benchmark: 13,800.0 downloads
  • S031 downloads change for GEMS-GER: A Machine Learning Benchmark Dataset of Long-Term Groundwater Levels in Germany with Meteorological Forcings and Site-Specific Environmental Features: -9,555.0 downloads
  • S032 downloads change for RyeAI/danish-asr-leaderboard: 5,367.0 downloads

What it means

What should someone building or evaluating AI systems do differently today?

medium confidence

Treat leaderboard rank or task completion as insufficient. Today’s captured releases emphasize cost-aware selection, end-to-end verification, evidence grounding, delivery completeness, and safety under realistic pressures rather than a single aggregate score.

Add checks that mirror actual deployment: compare quality against cost, verify outputs against observed evidence, test whether the whole job was completed, and inspect failures under operational or economic pressure. These recommendations come from newly released artifacts, not updates or public-attention signals in the feed. [E006, E010, E011, E012, E013]

Takeaway: Before adopting a model or agent, expand the acceptance test beyond headline accuracy. Record cost, correctness, evidence support, completion, and safety separately, then test in an environment resembling the intended workflow. The captured feed contains substantial benchmark and evaluation activity, but its category tags overlap and do not form exclusive groups.

Another reading: These releases are largely authors’ proposed benchmarks rather than independent replications. Their domains range from software optimization to web navigation, video generation, and economic environments, so the suggested checks may not transfer unchanged to every system. [E010, E011, E012, E013]

  • S003 records with event kind released: 305
  • S004 records with event kind updated: 256
  • S017 records tagged benchmark: 434 count (multi-label)
  • S018 records tagged evaluation: 341 count (multi-label)
  • S020 records tagged agentic: 104 count (multi-label)

What does today's evidence fail to show, and what would change the reading?

high confidence

The feed does not establish a field-wide contraction, a decisive new benchmark standard, or corroborated popularity movement. Recent captured observations were lower across benchmark, evaluation, dataset, agentic, and data-quality tags, but this remains a change within a keyword-filtered feed.

The category counts describe overlapping labels, not separate slices of the field. Download movements cover different full tracking spans and come from single sources, so they are not corroborated and must not be read as daily demand changes. The feed also misses unavailable connectors and is not representative of all AI work.

Takeaway: The reading would strengthen with restored connector coverage, repeated category movement across longer comparable windows, independent sources reporting the same metric movement, and external replications showing that newly released evaluations predict real deployment outcomes. Until then, interpret the observed declines and artifact activity as radar-specific evidence.

Another reading: Because the measurement windows are comparable and several overlapping tags moved lower together, the declines could reflect a genuine broad cooling in benchmark-related publication activity. The strongest limitation on that reading is that keyword filtering, overlapping categories, and incomplete connector coverage can produce coordinated changes without representing the wider field.

  • S024 tracked artifacts today seen by more than one data source: 13
  • S025 downloads change for lmarena-ai/leaderboard-dataset: 315,458.0 downloads
  • S026 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 39,722.0 downloads
  • S027 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 31,001.0 downloads
  • S028 downloads change for open-llm-leaderboard/requests: -28,441.0 downloads
  • S029 downloads change for lukakuma/web-trajectories-preference-raw: 23,620.0 downloads
  • S030 downloads change for alexshpunt/explicit-edit-benchmark: 13,800.0 downloads
  • S031 downloads change for GEMS-GER: A Machine Learning Benchmark Dataset of Long-Term Groundwater Levels in Germany with Meteorological Forcings and Site-Specific Environmental Features: -9,555.0 downloads
  • S032 downloads change for RyeAI/danish-asr-leaderboard: 5,367.0 downloads
  • S033 daily-average change in dataset observations: -35.43 observations per day
  • S034 daily-average change in benchmark observations: -34.71 observations per day
  • S035 daily-average change in evaluation observations: -26.71 observations per day
  • S036 daily-average change in agentic observations: -2.71 observations per day
  • S037 daily-average change in data_quality observations: -1.43 observations per day

Evidence sources

Where this came from, and what it does not cover

Built from the validated daily snapshot and its 635 evidence records.

Briefing model: gpt-5.6-sol.

It read 156 of 635 records.

This is a keyword-filtered feed, not a representative sample of artificial intelligence research. Only 156 of 635 evidence records were injected into this briefing, so unreviewed items may contain stronger releases. Brave Search, OpenReview, and Semantic Scholar were unavailable, and several attention connectors carried stale observations; no attention metrics were used in these findings.