Benchmark Radar™
RSS Contact Star —

Daily AI benchmark brief: 2026-10-26

464evidence observations
11sources represented
18public-attention signals

Daily briefing

  1. The new RVL-CDIP document-classification audit reports that about 10.5% of 400,000 documents have incorrect labels and 35.2% of the test set has a duplicate or near-duplicate in training. It also finds that page identification codes alone support 69% accuracy and that removing them reduces one model’s score by 11 points. Why it matters: If you use RVL-CDIP to select document models, its headline accuracy may partly measure label errors, repeated documents, and source-specific identifiers rather than transferable classification. The audit supports adding deduplication, corrected labels, identifier-masked tests, and external-document testing before making a model decision. Evidence: E011. High confidence.
  2. The new THReD Agent Benchmark evaluates an electronic health record preprocessing assistant with 141 tasks covering simple and multi-step transformations, failure recovery, requests where the agent should not act, schema changes, and prompt injection hidden in data. It uses synthetic health-record tables, expert reference pipelines, and outcome checks rather than patient data. Why it matters: If you are evaluating assistants that prepare clinical datasets, this artifact tests operational safety and recovery alongside successful preprocessing. It offers a way to compare context configurations without exposing patient records and avoids treating task completion alone as sufficient evidence of readiness. Evidence: E001. High confidence.
  3. The new “When Agents Fail” framework separates model behavior from the software harness that manages tools and execution. It measures valid completion, fit between a model and harness, fit across models, pairing-specific residuals, and failure modes. A separate controlled study of lightweight coding agents likewise evaluates alternative model-harness pairings rather than fixed leaderboard configurations. Why it matters: If you are choosing a self-hosted coding agent, a model score from one harness does not establish performance in another. These releases support testing the exact model-harness pair and recording whether failures come from task errors, abnormal termination, or the interaction between the two components. Evidence: E019, E010. High confidence.
  4. The new Async-RBench contains 200 executable tasks in eight scenarios where a main large language model agent continues working while delayed subagent results arrive under controlled task-state conditions. SWE-Shift independently tests coding agents when code, dependencies, runtime state, or requirements change during execution. Why it matters: Together, these artifacts expose a recurring pressure in this captured feed: long-running agents need evaluation under changing state, not only static tasks. If you are selecting an agent for concurrent or evolving work, this means checking whether it preserves valid work, revisits stale assumptions, and handles delayed results correctly before scoring the final outcome. Evidence: E021, E027. High confidence.
  5. The new Interactive Graphical User Interface Grounding benchmark changes interface testing from locating an already visible target to first revealing it and then clicking it. Its 601 tasks across 16 controlled environments cover targets that are off-screen, occluded, delayed, or exposed only after an interface state change. Why it matters: If you are comparing agents for real software interfaces, static screenshot localization misses whether an agent can recover visibility through scrolling, movement, dragging, waiting, or other interaction. This benchmark lets product teams separate target recognition from the preceding navigation needed to make the target actionable. Evidence: E016. High confidence.
  6. The new DMAIC-Bench tests whether an analytical agent changes its next action after reviewing intermediate evidence, a capability it calls self-restraint. Its 97 seeded tasks include paired counterfactual variants that require opposite actions at the same workflow position, distinguishing justified stopping or redirection from indiscriminate refusal. Why it matters: If you are evaluating agents for multi-stage analysis, a plausible self-critique does not show that evidence affected behavior. The paired design provides a way to test whether an agent continues when prerequisites hold and stops or redirects when they do not, rather than rewarding commentary or final-answer quality alone. Evidence: E023. High confidence.

What arrived

What benchmarks, datasets, or evaluation methods did the radar first see today?

medium confidenceNot enough evidence

The captured feed first observed a substantial cohort today. Notable releases included THReD for health-record preprocessing agents [E001], IGG for interactive interface grounding [E016], Async-RBench for asynchronous agents [E021], SWE-Shift for coding agents in changing workspaces [E027], and RxnExBench for chemical-reaction extraction [E025].

Other newly seen releases covered theoretical-physics reasoning with PHYRD [E015], customer-management agents with CRM-Agent-Bench [E029], multi-reference image generation with MDMR-Bench [E020], reasoning-based audio-visual grounding with RVG-Bench [E024], and open-set autonomous-driving perception with DriveOSS [E004].

Takeaway: Today’s captured arrivals span specialized scientific datasets, multimodal perception, and evaluations that test agent behavior during interaction or environmental change. This is a keyword-filtered radar feed, not a representative sample of AI work.

Another reading: “First observed” means new to this radar, not necessarily new to the world. The cohort also mixes releases with other event types: PTIT-CourseQA was updated [E079], while iPhoneBlur was discovered rather than released [E081]. The supplied evidence packet does not expose every first-observed artifact, so this is not a complete inventory.

  • S022 artifacts first observed by the radar today: 215

Which of today's arrivals document how they score an answer?

medium confidenceNot enough evidence

Confirmed examples include NoteSep, which provides scoring definitions [E009]; THReD, which uses expert reference pipelines and outcome checks [E001]; PHYRD, which provides reference solutions and subquestion rubrics [E015]; and the inquiry-learning benchmark, which includes human ratings and rating rubrics [E040].

Process-aware scoring also appears in When Agents Fail, which separates normal completion from task achievement and model–harness fit [E019]; CRM-Agent-Bench, which assesses outcomes, trajectories, operational viability, and policy compliance [E029]; and Aether, which supplies reference programs and treats safe refusal as correct in policy-trap tasks [E047].

Takeaway: The documented approaches range from checking against reference outputs to rubric-based human judgment and evaluating whether an agent followed a valid process, not merely whether its final response looked right [E001][E015][E019][E029][E040][E047].

Another reading: The summaries may omit scoring details found in full documentation, while some arrivals expose none at all; the domain-aware benchmarking record only says its data are forthcoming [E064]. Because the evidence packet does not include every first-observed artifact, no exhaustive list or registry-backed count is available.

What is still moving

Which artifacts the radar already tracked moved measurably, and over what span?

low confidenceNot enough evidence

The registry contains download-movement entries for named artifacts, including increases and a decline, but supplies no E-labeled evidence for them.

Because the evidence lists for these artifact statistics are empty, the artifacts and their individual spans cannot be named in a compliant answer.

Takeaway: Treat each registered change as cumulative movement across that artifact’s entire stated tracking span, not as a one-day change, and only within this keyword-filtered feed.

Another reading: The tracked-artifact table contains additional metric changes, so a broader ranking may exist; however, those changes lack registered statistics and E-labeled evidence needed for grounded reporting.

  • S025 downloads change for open-llm-leaderboard/requests: 59,113.0 downloads
  • S026 downloads change for vedangfake/chess-slm-benchmark: 19,463.0 downloads
  • S027 downloads change for hf-benchmarks/transformers: 13,951.0 downloads
  • S028 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 11,407.0 downloads
  • S029 downloads change for RyeAI/danish-asr-leaderboard: 6,540.0 downloads
  • S030 downloads change for vava22684/song-jury-leaderboard: -3,743.0 downloads
  • S031 downloads change for ToiTenBao/hallucination-evaluation-results: 2,399.0 downloads
  • S032 downloads change for hlido-eu/agent-benchmark: 1,817.0 downloads

Which of that movement is corroborated by more than one data source?

low confidenceNot enough evidence

No artifact-level movement can be reported as corroborated from the supplied registry and evidence packet.

The registry says some tracked artifacts appeared in more than one source, but that does not establish that multiple sources measured the same change.

Takeaway: A valid answer requires a movement statistic explicitly marked as corroborated and E-labeled evidence from multiple sources; those elements are missing here.

Another reading: The tracked-artifact metadata flags some records as corroborated, but corresponding registered movement statistics and E-labeled evidence are absent, so it cannot support a grounded artifact-level answer.

  • S024 tracked artifacts today seen by more than one data source: 14

What it means

What should someone building or evaluating AI systems do differently today?

medium confidence

Use today’s captured releases to widen evaluation checklists, not to swap models or declare winners.

For relevant systems, add checks for training-test overlap and misleading shortcuts; evaluate an agent’s model and execution harness together; and test interface tasks where the target is not initially visible. Today’s released audit and agent benchmarks identify these possible blind spots.

Takeaway: Run these checks on your own data and workflows before trusting headline benchmark results. The feed contains many overlapping benchmark, evaluation, and dataset records, but comparatively little explicitly tagged data-quality work.

Another reading: These artifacts are new releases rather than independent replications, and the feed shows no material category shift. They do not establish that the proposed checks improve outcomes across systems.

  • S017 records tagged benchmark: 289 count (multi-label)
  • S018 records tagged evaluation: 208 count (multi-label)
  • S019 records tagged dataset: 205 count (multi-label)
  • S020 records tagged agentic: 63 count (multi-label)
  • S021 records tagged data_quality: 12 count (multi-label)

What does today's evidence fail to show, and what would change the reading?

high confidence

Today’s captured feed does not show field-wide change, validated superiority, or corroborated adoption.

Category composition shows no material pattern. Download changes are cumulative across each artifact’s full registry-listed tracking span ending today, not daily changes, and each comes only from Hugging Face. Public attention observations are not releases or independent validation.

Takeaway: The reading would change with persistent comparable movement reported by multiple sources, independent replications on shared tasks, verified dataset audits, and restored coverage from unavailable connectors.

Another reading: The windows are comparable and the feed spans many sources, so the absence of a material pattern is meaningful within this capture. Keyword filtering and unavailable connectors still prevent generalization to the wider field.

  • S002 public attention observations captured today: 18
  • S024 tracked artifacts today seen by more than one data source: 14
  • S025 downloads change for open-llm-leaderboard/requests: 59,113.0 downloads
  • S026 downloads change for vedangfake/chess-slm-benchmark: 19,463.0 downloads
  • S027 downloads change for hf-benchmarks/transformers: 13,951.0 downloads
  • S028 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 11,407.0 downloads
  • S029 downloads change for RyeAI/danish-asr-leaderboard: 6,540.0 downloads
  • S030 downloads change for vava22684/song-jury-leaderboard: -3,743.0 downloads
  • S031 downloads change for ToiTenBao/hallucination-evaluation-results: 2,399.0 downloads
  • S032 downloads change for hlido-eu/agent-benchmark: 1,817.0 downloads
  • S033 daily-average change in benchmark observations: -45.86 observations per day
  • S034 daily-average change in evaluation observations: -14.0 observations per day
  • S035 daily-average change in agentic observations: -13.29 observations per day
  • S036 daily-average change in dataset observations: 3.43 observations per day
  • S037 daily-average change in data_quality observations: -0.86 observations per day

Evidence sources

Where this came from, and what it does not cover

Built from the validated daily snapshot and its 464 evidence records.

Briefing model: gpt-5.6-sol.

It read 123 of 464 records.

This is a keyword-filtered feed, not a representative sample of artificial intelligence work. Only 123 of 464 captured evidence records were injected into this briefing, although none of those 123 were dropped for size. Brave, OpenReview, and Semantic Scholar were unavailable, so relevant releases from those connectors may be missing.