Benchmark Radar™
RSS Contact Star —

Daily AI benchmark brief: 2026-09-23

510evidence observations
11sources represented
16public-attention signals

Daily briefing

  1. A new B3DB audit tests whether stereoisomers inflate blood–brain barrier prediction scores because common two-dimensional molecular representations map different three-dimensional arrangements to the same input, allowing effective duplicates across scaffold-separated train and test partitions. It compares conventional canonical-string deduplication with representation-aware curation across 7,807 records. [E002] Why it matters: If you evaluate molecular models that cannot distinguish stereoisomers, a scaffold split alone may not establish independence. This artifact makes model-visible identity, rather than chemical record identity, a concrete criterion for dataset deduplication and split design. Evidence: E002. High confidence.
  2. The new post-training delivery benchmark in “Trains but Doesn't Learn” evaluates an agent across ten governed stages, including a budget, human approval, reproducibility requirements, and platform-recorded scoring. It explicitly targets deliveries where training loss falls but the resulting model does not improve. [E016] Why it matters: If you are selecting an agent to fine-tune and deploy models, an endpoint score cannot show whether it used the intended data, respected controls, or delivered the tested model. This benchmark shifts the decision from choosing the best optimizer to choosing a system that can complete an auditable delivery process. Evidence: E016. High confidence.
  3. The new stale benchmark evaluates parallel coding agents by testing each patch separately and then testing the merged pair, counting failures caused specifically by their combination. Its tiers include controlled interface changes, mined pull-request pairs, and constructed tasks using real Django software helpers. [E015] Why it matters: If you are evaluating multiple agents that edit one codebase concurrently, individual patch pass rates miss incompatible assumptions between otherwise successful changes. This paired procedure adds a direct measure of semantic coordination to the suite-selection decision. Evidence: E015. High confidence.
  4. The new KEX-bench separates vulnerability discovery from exploit construction. Its 45 tasks cover 40 Linux and Windows vulnerabilities and require agents to produce specific exploit primitives, such as an address leak or arbitrary memory write, inside isolated virtual machines with deterministic success checks. [E014] Why it matters: If you are assessing coding agents for defensive security, finding or describing a bug is not evidence that an agent can operationalize it. KEX-bench provides a distinct capability boundary that can change threat assessments and access-control decisions for security agents. Evidence: E014. High confidence.
  5. The new SSP-Bench generates safety, security, and privacy evaluation cases on demand rather than relying only on a fixed test set. Its design grounds labels in external sources, validates service-specific scope, varies wording while preserving meaning, and uses multiple models to calibrate difficulty. [E008] Why it matters: If you are choosing a safety suite for repeated releases, fixed questions can become familiar to models and hide sensitivity to rephrasing. This design offers a way to test the same policy concepts with fresh language while retaining explicit controls over labels, scope, and difficulty. Evidence: E008. High confidence.
  6. The new AgentSoD-P2P benchmark treats separation of duties as more than assigning maker and checker roles to different identities. It uses 160 fixed procurement proposals and a controlled design that varies whether the checker shares the maker’s model dependence, then measures unsafe approvals and disruption of legitimate workflows. [E003] Why it matters: If you are designing an artificial intelligence approval workflow, two nominally separate agents may not provide independent assurance when they rely on the same model. This benchmark makes model composition and correlated failure part of the authorization evaluation rather than an architectural assumption. Evidence: E003. Medium confidence.
  7. The new SWE-Serve benchmark contains 53 repository-scale tasks for production model-serving engineering, where an agent may need coordinated changes across model support, runtime execution, and public interfaces. It targets a gap between general software-repair suites and narrow kernel or performance tests. [E010] Why it matters: If you are selecting a coding agent to maintain inference infrastructure, success on isolated code fixes does not establish that it can implement a serving feature across several system layers. SWE-Serve supplies a task boundary closer to that deployment decision. Evidence: E010. High confidence.
  8. The new WebArxiv benchmark freezes a snapshot of the arXiv website and supplies 510 time-invariant research-navigation tasks with unique deterministic answers. It preserves web interaction while avoiding score changes caused by a live site’s evolving content and structure. [E017] Why it matters: If you compare web agents across models or dates, live-site drift can make score differences inseparable from environmental changes. A static snapshot supports reproducible comparisons, though it measures interaction with the frozen environment rather than adaptation to current websites. Evidence: E017. High confidence.

What arrived

What benchmarks, datasets, or evaluation methods did the radar first see today?

high confidenceNot enough evidence

In this captured feed, first-seen release records included SSP-Bench, SWE-Serve, VIS-GEN, KEX-bench, WebArxiv, and an expert-annotated legal-argument corpus. The radar also first encountered a TabSets update and discovery signals for RoboFollow and Taste-Bench.

The evaluation designs include generating safety cases on demand, checking exploit success with a fixed verifier, preserving websites as unchanging test environments, testing whether separately valid software patches fail together, and judging generated graphics against task-specific rubrics.

Takeaway: Today’s keyword-filtered feed surfaced varied evaluation targets and methods, but this is a selective catalog rather than a representative field sample. With no certified comparison window, these arrivals do not establish that any topic is becoming more common.

Another reading: This is not a complete inventory because the evidence packet contains selected records rather than every first-observed artifact. First observed by the radar also does not mean newly created today: TabSets was an update, while RoboFollow and Taste-Bench were discovery signals rather than captured release records.

  • S022 artifacts first observed by the radar today: 234

Which of today's arrivals document how they score an answer?

high confidenceNot enough evidence

Clear examples in the captured packet include the enterprise-authorization benchmark’s deterministic gold decisions, KEX-bench’s deterministic success verifier, stale’s combination-only test failures, the delivery benchmark’s oracle stage scores, the geoparser results’ exact-match and gold-span rules, and RULER’s multi-axis rubric judge.

These records explain what turns an output into a score: compare it with a fixed correct decision, run an automatic success check, rerun tests after combining patches, score recorded completion at each stage, compare exact text spans, or apply a written judging checklist.

Takeaway: These artifacts provide more scoring detail than records that merely mention metrics or leaderboards. Most cited examples are release records; RULER entered this feed as a discovery signal, so it should not be described as a new release.

Another reading: The supplied summaries still omit full formulas, thresholds, weighting rules, and judge prompts for several artifacts. Consequently, they identify documented scoring approaches but do not establish that every method is fully reproducible or enumerate every qualifying arrival in the feed.

  • S022 artifacts first observed by the radar today: 234

What is still moving

Which artifacts the radar already tracked moved measurably, and over what span?

low confidenceNot enough evidence

The attached registry entries identify the artifacts with highlighted measurable movement. Most record cumulative download changes, while one records cumulative star movement; each has its own start-to-end tracked span.

These changes accumulated across each artifact’s full registered observation period, not during a single day. One attached download measure declined, while the other attached measures increased.

Takeaway: The statistics provide artifact names, metric direction, and tracked spans, but the packet supplies no artifact-level evidence identifiers with which to validate them.

Another reading: Missing artifact-level evidence identifiers prevent independent confirmation of these measurements, and no certified comparison window is available for broader trend interpretation.

  • S025 downloads change for hf-benchmarks/transformers: 23,353.0 downloads
  • S026 downloads change for IntelligenceLab/LHTB-leaderboard: -15,567.0 downloads
  • S027 downloads change for lmarena-ai/leaderboard-dataset: 14,623.0 downloads
  • S028 downloads change for RoboDojo-Benchmark/RoboDojo: 13,218.0 downloads
  • S029 downloads change for vedangfake/chess-slm-benchmark: 9,644.0 downloads
  • S030 downloads change for hffordata/h3-fewstep-benchmark-20260920: 5,823.0 downloads
  • S031 stars change for career-ops-hq/career-ops: 2,793.0 stars
  • S032 downloads change for hf-audio/open-asr-leaderboard-results: 2,761.0 downloads

Which of that movement is corroborated by more than one data source?

high confidenceNot enough evidence

None of the attached movement statistics is marked as corroborated by more than one source.

Seeing an artifact in multiple feeds is not enough. Corroboration requires more than one source to report the same metric movement, and the registry does not provide such a movement statistic here.

Takeaway: Do not treat multisource artifact sightings as confirmation of metric movement. A registered per-metric movement with multiple reporting sources is missing.

Another reading: The tracked-artifact packet marks some other records as corroborated, but their metric movements lack registered movement statistics and evidence identifiers, so they cannot answer this question under the grounding rules.

  • S024 tracked artifacts today seen by more than one data source: 7
  • S025 downloads change for hf-benchmarks/transformers: 23,353.0 downloads
  • S026 downloads change for IntelligenceLab/LHTB-leaderboard: -15,567.0 downloads
  • S027 downloads change for lmarena-ai/leaderboard-dataset: 14,623.0 downloads
  • S028 downloads change for RoboDojo-Benchmark/RoboDojo: 13,218.0 downloads
  • S029 downloads change for vedangfake/chess-slm-benchmark: 9,644.0 downloads
  • S030 downloads change for hffordata/h3-fewstep-benchmark-20260920: 5,823.0 downloads
  • S031 stars change for career-ops-hq/career-ops: 2,793.0 stars
  • S032 downloads change for hf-audio/open-asr-leaderboard-results: 2,761.0 downloads

What it means

What should someone building or evaluating AI systems do differently today?

medium confidence

Within this keyword-filtered feed, today’s releases highlight evaluation blind spots involving duplicate-like data across splits, sensitivity to equivalent wording, production-scale agent work, interactions between merged outputs, and confidence transfer between agents.

Before accepting a benchmark result, check for hidden overlap between training and testing data, harmless wording changes, realistic end-to-end work, failures that appear only when separate outputs are combined, and whether confidence estimates remain reliable on a different agent.

Takeaway: Add these checks to release gates rather than relying on one leaderboard score. Preserve run-level evidence, test deployment-shaped tasks, and report failures separately so aggregate results cannot conceal contamination, coordination, reliability, or reproducibility problems.

Another reading: These are newly released, mostly self-described artifacts rather than independent proof that existing evaluations are broadly defective. The coordination study’s mined real-world portion found almost no interference after its grading correction, so that risk may depend heavily on task construction.

  • S017 records tagged benchmark: 331 count (multi-label)
  • S018 records tagged evaluation: 243 count (multi-label)
  • S019 records tagged dataset: 212 count (multi-label)
  • S020 records tagged agentic: 76 count (multi-label)
  • S021 records tagged data_quality: 5 count (multi-label)

What does today's evidence fail to show, and what would change the reading?

high confidenceNot enough evidence

This captured feed does not establish field-wide growth, adoption, benchmark quality, or model improvement. There is no certified comparison window, and only a small part of today’s tracked artifacts appeared through more than one data source.

Differences from other days may reflect collection changes rather than changes in the field. A release label means an artifact was published, an update is not a new release, and public discussion shows attention rather than technical effectiveness.

Takeaway: The reading would change with stable connector coverage, an unchanged taxonomy, enough comparable history, independent reproductions, direct baseline results, and corroborated measurements from multiple sources. Until then, treat today’s material as candidates for inspection rather than evidence of a broader shift.

Another reading: The captured feed still offers concrete practices worth examining: reproducible website snapshots, deterministic success checks, and repositories designed to expose auditable evaluation evidence. Those artifacts may be useful even though the feed cannot establish prevalence or quality across the field.

  • S002 public attention observations captured today: 16
  • S003 records with event kind released: 245
  • S004 records with event kind updated: 213
  • S005 records with event kind discovered: 52
  • S022 artifacts first observed by the radar today: 234
  • S023 artifacts seen today that the radar had already tracked: 276
  • S024 tracked artifacts today seen by more than one data source: 7

Evidence sources

Where this came from, and what it does not cover

Built from the validated daily snapshot and its 510 evidence records.

Briefing model: gpt-5.6-sol.

It read 138 of 510 records.

This briefing examined 138 selected evidence records from a 510-record keyword-filtered corpus, so it does not represent a full reading of the captured feed or the wider artificial intelligence field. OpenReview, Semantic Scholar, and Brave were unavailable today. The collection also lacks enough comparable history to infer category-share movement, so the findings describe individual releases rather than field-wide trends.