Benchmark Radar™
RSS Contact Star —

Daily AI benchmark brief: 2026-10-15

899evidence observations
14sources represented
16public-attention signals

Daily briefing

  1. A new audit found that all 761 traceable tumor test images in a widely used brain magnetic resonance imaging benchmark came from patients also represented in training. Patient-disjoint cross-validation—keeping each patient wholly within one split—reduced reported overall accuracy from roughly 99% to 95%, while simply removing pixel duplicates did not produce the same correction. Why it matters: If you evaluate medical imaging models, image-level deduplication is insufficient when multiple images can come from one patient. Model-selection claims from this benchmark need patient-disjoint splits to measure generalization to unseen patients rather than recognition of patient-linked features. Evidence: E045. High confidence.
  2. EvoProbe is a new framework that continually regenerates large language model agent tasks using diagnostic feedback. It scores verifiable subtasks, evaluates complete action sequences with evidence-supported judges, and extracts phase-level bottlenecks rather than relying only on final success rates. GraceBench and HICO-Bench independently reflect the same captured-feed pressure toward diagnosing trajectories and verifying resulting plans. Why it matters: If you maintain an agent evaluation suite, EvoProbe offers a design for updating tasks around observed failure modes instead of adding undirected examples. Its phase-level evidence can distinguish whether a product change fixed planning, tool execution, or another stage even when the aggregate success score barely moves. Evidence: E001, E159, E125. High confidence.
  3. Doc2Harness is a new two-stage benchmark that evaluates the software harness around a coding agent—the tools, interfaces, and control logic that let the model work—rather than holding that harness fixed. Its first stage asks a model to construct a complete candidate harness from an interface contract and a development document covering twelve frontier harnesses. Why it matters: If you are choosing a coding-agent stack, this separates model capability from harness engineering. It enables comparisons of whether a model can build the surrounding execution system, rather than attributing all downstream task success to the model used inside a preselected harness. Evidence: E012. High confidence.
  4. MMA-SafetyBench is a new safety benchmark for multimodal agents that inserts deceptive visual elements, such as forged system alerts, into an agent’s observation stream. It contains 1,083 adversarial trajectories across web, desktop, mobile, document, and video tasks, targeting visual semantic hijacking: interface content that diverts an agent from its authorized task. Why it matters: If your agent acts through visual interfaces, text-only prompt-injection tests do not cover this attack path. This benchmark lets a safety review test whether the agent follows authorization boundaries when malicious instructions appear as meaningful on-screen content during a multi-step task. Evidence: E008. High confidence.
  5. RCA-Bench is a new benchmark for whether an agent chooses an appropriate autonomy level. It separates substantive decisions reserved for humans from authorization to execute, then grades responses as ACT, CONFIRM, or ESCALATE across 60 expert-validated organizational scenarios and six domains. Its release includes 1,440 responses from four models under baseline and policy-guided prompting. Why it matters: If you are setting deployment permissions for workplace agents, this evaluates the delegation decision itself rather than only task correctness. The three-way outcome can reveal whether a policy prompt reduces unauthorized action, causes excessive escalation, or appropriately requests confirmation. Evidence: E006. High confidence.
  6. BenchChecker is a new contamination-audit framework for bug-fixing benchmarks. It uses textual model outputs and public repository history to look for evidence that benchmark patches may have appeared in training, and its source description says the method works with both open- and closed-source models. Why it matters: If you compare coding models whose training data are unavailable, this provides a possible audit path without inspecting model weights or private corpora. Contamination findings can determine whether a high repair score supports deployment selection or mainly reflects exposure to persistent repository patches. Evidence: E024. Medium confidence.
  7. J-tau is a new Japanese benchmark for tool-using customer-support agents, localized from the English-centric tau-bench through translation, adaptation of locale-specific entities such as addresses and currencies, and human review. Its construction also identified and fixed two flaws in the original benchmark. Why it matters: If you are selecting an agent for Japanese customer support, a translated English test may preserve unrealistic entities and interaction assumptions. J-tau provides a locale-adapted alternative, while the reported fixes mean scores should not be treated as directly interchangeable with results from the uncorrected source benchmark. Evidence: E013. High confidence.

What arrived

What benchmarks, datasets, or evaluation methods did the radar first see today?

high confidenceNot enough evidence

The captured feed first saw a broad mix of artifacts today, including EvoProbe for evolving agent evaluations, CausalDS for causal data-science agents, Doc2Harness for coding-agent harnesses, OpenTWBench for Taiwan knowledge, ListQA for list-form answers, and BenchChecker for contamination detection.

Other first-seen releases covered cross-platform code migration, Japanese tool use, emotional user interactions, garment reconstruction, outdoor low-light video, Persian hate-speech analysis, and code-switching. Separately, the radar discovered RuleTrek for rule reasoning and HICO-Bench for checking whether social agents reach workable, conversation-supported plans.

Takeaway: The first-seen set emphasizes executable agent tasks, specialized languages and domains, benchmark reliability, and richer answer formats. These are examples from a keyword-filtered feed, and the category tags overlap rather than dividing the arrivals into exclusive groups.

Another reading: First observed by this radar does not establish first publication or novelty to the field. RCA-Bench, for example, carried an earlier publication timestamp, and incomplete source coverage means the supplied packet cannot support a complete inventory of every first-seen artifact.

  • S025 artifacts first observed by the radar today: 484

Which of today's arrivals document how they score an answer?

high confidenceNot enough evidence

The clearest cases are EvoProbe, which combines verifiable subtask outcomes with judged trajectories; the Power Systems Agent Benchmark, which provides deterministic evaluators; CreatorMemBench, which supplies a strict semantic protocol and aggregation script; and UniBench-NLP, which includes judge verdicts, human ratings, and scripts that reproduce reported results.

HICO-Bench checks whether both agents recorded a workable plan and whether the conversation supports it. AgentStripes converts capability ratings and evidence strength into credits and an aggregate score. Into the Unknown builds rubrics from scientist-provided requirements, while JockeyBench reports forecast quality, task-specific usefulness, and retrospective outcomes separately.

Takeaway: These arrivals expose more than a final leaderboard value: they describe checks, judging layers, aggregation, or reproducibility machinery. The registry has no dedicated count of arrivals with documented scoring, so a complete total or share cannot be reported.

Another reading: A summary saying that evaluators or scripts exist does not prove that every scoring detail is public. Full judge prompts, tie handling, failure rules, and rubric examples may require inspecting the linked artifacts, so this is an evidence-backed shortlist rather than a complete audit.

What is still moving

Which artifacts the radar already tracked moved measurably, and over what span?

high confidenceNot enough evidence

The registry contains named download-movement statistics with full tracked spans, but none has an attached evidence record.

Each registered change is cumulative across the span stated in its statistic, not a one-day change. Artifact-specific reporting is not grounded because the supplied evidence IDs are empty.

Takeaway: The captured feed shows measurable movement entries, but identifying the artifacts as verified findings requires artifact-level evidence links that were not supplied.

Another reading: A looser reading could treat the named registry entries as sufficient by themselves, but that would violate the requirement to cite evidence for each specific artifact.

  • S028 downloads change for lmarena-ai/leaderboard-dataset: 79,298.0 downloads
  • S029 downloads change for vedangfake/chess-slm-benchmark: 22,699.0 downloads
  • S030 downloads change for huggingface-projects/drlc-leaderboard-data: 18,880.0 downloads
  • S031 downloads change for hf-benchmarks/transformers: 16,942.0 downloads
  • S032 downloads change for optimum-benchmark/cpu: -13,541.0 downloads
  • S033 downloads change for Rapidata/svg-benchmark: 8,712.0 downloads
  • S034 downloads change for AlphaDojo/dojo_benchmark_kline: 4,412.0 downloads
  • S035 downloads change for NoeFlandre/benchmark-llms-landuse-relevance: 3,790.0 downloads

Which of that movement is corroborated by more than one data source?

high confidenceNot enough evidence

None of the registered movement statistics is marked as corroborated; each reports only one source.

Seeing an artifact in multiple sources does not prove that multiple sources measured the same change. The registry separately warns against making that inference.

Takeaway: No artifact movement can be reported as corroborated from the registered statistics in this keyword-filtered captured feed.

Another reading: Some tracked-artifact metadata carries corroboration flags, but those movements lack corresponding registered statistics and artifact-level evidence IDs, so they cannot be reported under the grounding rules.

  • S027 tracked artifacts today seen by more than one data source: 37
  • S028 downloads change for lmarena-ai/leaderboard-dataset: 79,298.0 downloads
  • S029 downloads change for vedangfake/chess-slm-benchmark: 22,699.0 downloads
  • S030 downloads change for huggingface-projects/drlc-leaderboard-data: 18,880.0 downloads
  • S031 downloads change for hf-benchmarks/transformers: 16,942.0 downloads
  • S032 downloads change for optimum-benchmark/cpu: -13,541.0 downloads
  • S033 downloads change for Rapidata/svg-benchmark: 8,712.0 downloads
  • S034 downloads change for AlphaDojo/dojo_benchmark_kline: 4,412.0 downloads
  • S035 downloads change for NoeFlandre/benchmark-llms-landuse-relevance: 3,790.0 downloads

What it means

What should someone building or evaluating AI systems do differently today?

medium confidence

In this captured, keyword-filtered feed, agentic observations had a higher recent-week daily average than in the prior week, while evaluation observations had a lower average. New releases address autonomy calibration, visual-interface attacks, harness engineering, localization, emotional interactions, and cross-operating-system migration (E006, E008, E012, E013, E014, E015).

Audit whether your tests cover the ways deployed agents actually fail. Consider adding shadow tests for unsafe autonomy, misleading screen content, supporting software around the model, local language and culture, difficult user interactions, and platform migration before changing production gates (E006, E008, E012, E013, E014, E015).

Takeaway: Keep existing baselines, then pilot only the newly released benchmark most closely matching your deployment risk. Treat its results as preliminary until you verify the tasks, scoring, contamination controls, and reproducibility in your own environment.

Another reading: The aggregate signal does not establish urgency: benchmark and data-quality observations had lower recent-week daily averages, and the releases provide no demonstrated superiority over an organization’s existing evaluation suite. The strongest reading may therefore be to review coverage without changing gates yet.

  • S003 records with event kind released: 567
  • S036 daily-average change in evaluation observations: -36.14 observations per day
  • S037 daily-average change in agentic observations: 6.86 observations per day
  • S039 daily-average change in benchmark observations: -2.43 observations per day
  • S040 daily-average change in data_quality observations: -0.71 observations per day

What does today's evidence fail to show, and what would change the reading?

high confidence

This captured feed does not show that today’s releases improve real deployment outcomes, outperform established evaluations, or have broad adoption. The reported download movements cover differing full tracked spans, are not daily changes, and come from a single source, so none is corroborated.

Release notices and download counts show availability or interest, not whether a benchmark is reliable, hard to game, relevant to production, or predictive of failures. The feed is keyword-filtered and therefore cannot establish a field-wide shift.

Takeaway: The reading would strengthen with independent replications, comparable head-to-head results, disclosed task and scoring audits, contamination checks, and movement confirmed by multiple sources. A sustained change across comparable windows would also support a broader trend interpretation.

Another reading: The feed still has discovery value: most captured records were releases, and some tracked artifacts were seen by more than one source. That breadth can identify candidates for review, although multiple sightings do not validate quality or corroborate a particular metric.

  • S003 records with event kind released: 567
  • S027 tracked artifacts today seen by more than one data source: 37
  • S028 downloads change for lmarena-ai/leaderboard-dataset: 79,298.0 downloads
  • S029 downloads change for vedangfake/chess-slm-benchmark: 22,699.0 downloads
  • S030 downloads change for huggingface-projects/drlc-leaderboard-data: 18,880.0 downloads
  • S031 downloads change for hf-benchmarks/transformers: 16,942.0 downloads
  • S032 downloads change for optimum-benchmark/cpu: -13,541.0 downloads
  • S033 downloads change for Rapidata/svg-benchmark: 8,712.0 downloads
  • S034 downloads change for AlphaDojo/dojo_benchmark_kline: 4,412.0 downloads
  • S035 downloads change for NoeFlandre/benchmark-llms-landuse-relevance: 3,790.0 downloads
  • S036 daily-average change in evaluation observations: -36.14 observations per day
  • S037 daily-average change in agentic observations: 6.86 observations per day
  • S038 daily-average change in dataset observations: 3.71 observations per day
  • S039 daily-average change in benchmark observations: -2.43 observations per day
  • S040 daily-average change in data_quality observations: -0.71 observations per day

Evidence sources

Where this came from, and what it does not cover

Built from the validated daily snapshot and its 899 evidence records.

Briefing model: gpt-5.6-sol.

It read 189 of 899 records.

This briefing covers a keyword-filtered, nonrepresentative feed. Only 189 of 899 corpus evidence records were injected for analysis, so these findings do not represent a full reading of the captured corpus. Brave and OpenReview were unavailable, and several attention collectors had failed or unread requests; no field-wide prevalence claim follows from this evidence.