Benchmark Radar™
RSS Contact Star —

Daily AI benchmark brief: 2026-10-28

573evidence observations
13sources represented
15public-attention signals

Daily briefing

  1. A new release in this feed, OpenGray, evaluates whether radiotherapy-planning agents make the correct decision, not merely whether they produce a favorable dose plan. Its known-answer tasks include incompatible treatment goals, where an agent should recognize that no valid plan exists, and a feasible control with a stored plan satisfying every hard goal. Why it matters: If you evaluate clinical planning agents, OpenGray lets you test whether an agent appropriately requests review or stops, rather than rewarding it for optimizing an impossible specification. That distinction can change which system appears safe enough for further validation. Evidence: E001. High confidence.
  2. The newly released EffectWorld represents each agent tool by what it reads, writes, and changes, including whether repeated calls are safe. An executable verifier can then check tool-use traces for unsafe effects without relying on human labels or another model as judge. Why it matters: If you build tool-using agents, this supports evaluation of actions such as duplicate writes or deletion without confirmation even when the final answer looks correct. It also offers machine-checkable training signals for reducing wasteful or unsafe tool sequences. Evidence: E006. High confidence.
  3. LiveSearch, newly discovered by this feed, stores financial-search questions as executable verifier programs. The question remains fixed, but its current answer is generated through calls to a professional financial data service rather than frozen when the benchmark is authored. Why it matters: If you are choosing an evaluation for financial search agents, this design tests current retrieval and analysis instead of recall of stale answers. It changes the benchmark dependency from a static answer file to executable access to the underlying data service. Evidence: E090. High confidence.
  4. FeatureArc, newly discovered by this feed, evaluates coding agents across successive extensions of the same feature. Each stage builds on the agent’s own earlier code, while executable checks test both new requirements and continuing compatibility. Why it matters: If your product relies on agents maintaining software over time, a one-request benchmark can miss regressions caused by earlier design choices. FeatureArc provides a way to compare agents on cumulative maintenance rather than isolated patch completion. Evidence: E095. High confidence.
  5. A new release introduces Bipartite Attribution, or BiPA, for auditing agent skill optimizers. It reruns an optimizer’s acceptance rule on candidates identical to the starting procedure, revealing gains produced by noisy selection alone, and traces the extra input tokens associated with accepted edits. Why it matters: If you evaluate systems that automatically rewrite an agent’s instructions, this separates actual procedural improvement from chance acceptance on a small development set. It also makes token cost part of the comparison rather than treating task success as the only outcome. Evidence: E041. High confidence.
  6. A new audit in this feed examines 51 multilingual benchmarks covering 242 datasets and 219 languages. It reports that 36% of represented languages occur in only one benchmark, while some regions receive almost no coverage and lower-resource languages span far fewer task categories. Why it matters: If you are selecting a multilingual evaluation suite, language count alone can overstate coverage. The audit supports comparing geographic representation and task variety before using aggregate multilingual scores to make model-selection claims. Evidence: E003. High confidence.
  7. The new CogArena benchmark runs agents through ten behavioral experiments in a standard web browser, using the same visual stimuli and response interfaces encountered by human participants instead of converting the experiments into text prompts. Why it matters: If you study model behavior using human experimental paradigms, CogArena lets you test whether conclusions survive the visual interface and timing conditions. This can change whether a text-only result is treated as evidence about behavior in the original experiment. Evidence: E002. High confidence.
  8. The newly released ETC-Bench evaluates transcript canonicalization: converting different written forms of the same speech into one comparison form when the audio cannot determine the difference. It covers 181 phenomenon types across 12 languages and tests both normalization and preservation of meaning. Why it matters: If you compare speech-recognition systems, this can separate genuine recognition errors from harmless spelling, formatting, or transcription conventions. That may change model rankings when conventional exact-text scoring penalizes equivalent transcripts. Evidence: E013. High confidence.

What arrived

What benchmarks, datasets, or evaluation methods did the radar first see today?

medium confidenceNot enough evidence

Notable first-seen releases included OpenGray for radiotherapy-planning decisions (E001), CogArena for browser-based behavioral experiments (E002), EffectWorld for tool-use safety (E006), a contradiction-aware retrieval benchmark (E008), SOMA for body-motion understanding (E009), Cordon’s cloud-control scenarios (E019), and SYNERGY+ expert screening data (E020). Separately, the radar newly discovered LiveSearch (E090), FeatureArc (E095), AutoVend (E097), and InfraDev (E152); discovery does not establish a release today.

The captured arrivals span medical planning, interactive agents, safe tool use, search, video understanding, cloud operations, software development, and expert-labeled review data. Some were presented as new releases, while others were older or external artifacts that this radar happened to encounter for the first time today (E001, E002, E006, E008, E009, E019, E020, E090, E095, E097, E152).

Takeaway: This keyword-filtered feed first observed many artifacts today, but the supplied evidence packet contains only a subset. The examples show unusually broad evaluation coverage rather than one dominant benchmark theme, and “first observed” should not be read as “first published” or as representative of the wider field.

Another reading: The selected packet is not sufficient for a complete inventory of all first-seen artifacts. Some sources were unavailable or rate-limited, and several records labeled as discovered were published before today, so this is primarily a collection-timing signal rather than a comprehensive release map.

  • S024 artifacts first observed by the radar today: 236

Which of today's arrivals document how they score an answer?

medium confidenceNot enough evidence

The clearest documented scorers are OpenGray’s known-correct decisions (E001), EffectWorld’s replay verifier (E006), Cordon’s expected decisions and machine-checkable final states (E019), Addis Scribe’s shared word-error scorer (E052), HyperLogic’s execution-derived answers (E070), HoldoutBench’s program grading (E072), and LiveSearch’s executable verifier code (E090). FeatureArc uses executable checks (E095), while Req2Test-Bench (E106), UnityBench-Mini3D (E122), and InfraDev (E152) describe execution tests or hidden probes.

These arrivals do more than ask a model or agent for an output. They compare it with a known decision, run code to compute the answer, inspect the resulting system state, replay the action trace, calculate transcription errors, or execute tests against the produced software (E001, E006, E019, E052, E070, E072, E090, E095, E106, E122, E152).

Takeaway: Executable or rule-based grading is the strongest pattern in the documented subset. LiveSearch, HoldoutBench, HyperLogic, and the software-agent benchmarks reduce reliance on subjective model judging, while OpenGray, Cordon, and Addis Scribe expose reference decisions or a consistent scorer (E001, E019, E052, E070, E072, E090, E095, E106, E122, E152).

Another reading: The evidence consists mainly of titles and summaries, not inspected scoring implementations. A claim of executable grading does not establish that the verifier fully captures correctness, and hidden probes in InfraDev are not independently auditable from the supplied record (E152). The packet also cannot establish an exhaustive list of scoring-documented arrivals.

  • S024 artifacts first observed by the radar today: 236

What is still moving

Which artifacts the radar already tracked moved measurably, and over what span?

high confidenceNot enough evidence

The registry contains artifact-level download movements, each cumulative across the full tracked span shown in its statistic rather than a one-day change. However, the packet provides no E-prefixed evidence identifiers for those artifacts.

The computed statistics identify the moving artifacts and their observation windows, but the required source citations are absent. A compliant artifact-by-artifact list therefore cannot be produced from this packet.

Takeaway: Supply evidence records with E-prefixed identifiers linked to each movement statistic. Until then, the registry values and spans are computable but not fully citable under the radar’s grounding rules.

Another reading: The statistic registry itself names the artifacts, gives their spans, and records their download changes, so the computation is present. The limitation is citation completeness, not the absence of measured movement.

  • S027 downloads change for lmarena-ai/leaderboard-dataset: 182,705.0 downloads
  • S028 downloads change for open-llm-leaderboard/requests: 60,232.0 downloads
  • S029 downloads change for huggingface-projects/drlc-leaderboard-data: 39,135.0 downloads
  • S030 downloads change for vedangfake/chess-slm-benchmark: 23,218.0 downloads
  • S031 downloads change for hf-benchmarks/transformers: 16,434.0 downloads
  • S032 downloads change for alexshpunt/explicit-edit-benchmark: 15,442.0 downloads
  • S033 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 13,641.0 downloads
  • S034 downloads change for RyeAI/danish-asr-leaderboard: 6,710.0 downloads

Which of that movement is corroborated by more than one data source?

high confidence

None of the movement statistics covered above is corroborated by more than one data source; each is sourced only from Hugging Face.

Another source did not independently report the same download movement for any of those entries. Seeing an artifact in multiple feeds would not by itself confirm its metric change.

Takeaway: Treat all listed download movements as single-source measurements within this keyword-filtered captured feed, not as independently confirmed changes or representative field-wide signals.

Another reading: Some tracked artifacts were seen by multiple sources, but the registry explicitly distinguishes an independent artifact sighting from corroboration of the same metric. That competing reading does not validate these movements.

  • S026 tracked artifacts today seen by more than one data source: 30
  • S027 downloads change for lmarena-ai/leaderboard-dataset: 182,705.0 downloads
  • S028 downloads change for open-llm-leaderboard/requests: 60,232.0 downloads
  • S029 downloads change for huggingface-projects/drlc-leaderboard-data: 39,135.0 downloads
  • S030 downloads change for vedangfake/chess-slm-benchmark: 23,218.0 downloads
  • S031 downloads change for hf-benchmarks/transformers: 16,434.0 downloads
  • S032 downloads change for alexshpunt/explicit-edit-benchmark: 15,442.0 downloads
  • S033 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 13,641.0 downloads
  • S034 downloads change for RyeAI/danish-asr-leaderboard: 6,710.0 downloads

What it means

What should someone building or evaluating AI systems do differently today?

medium confidence

Use today’s captured releases to broaden local testing, especially for consequential decisions, interactive agents, tool side effects, and multilingual coverage. OpenGray tests planning decisions under feasible and contradictory conditions; CogArena uses interactive browser experiments; EffectWorld checks tool effects; and a multilingual benchmark audit identifies coverage and rigor concerns (E001; E002; E006; E003).

Do not rely only on an overall score. Add tests that reveal whether a system recognizes impossible requirements, behaves safely while using tools, works through the real interface users encounter, and performs adequately across relevant languages. These are newly released evaluation approaches in this captured feed, not proof that any particular model or method is better (E001; E002; E003; E006).

Takeaway: Treat today’s feed as a source of test designs, not as a buying or deployment signal. Before promoting a system, run locally relevant failure cases, preserve executable checks where available, and document language, interface, and safety coverage. The feed contains substantial released and updated material, while its category tags overlap and should not be read as separate market segments.

Another reading: Existing evaluation suites may already cover these risks, and the packet provides descriptions rather than independent validation of the new releases. Adopting them immediately could add cost without changing a deployment decision, particularly when their tasks, interfaces, or domains do not match the intended system.

  • S003 records with event kind released: 247
  • S004 records with event kind updated: 217
  • S019 records tagged benchmark: 406 count (multi-label)
  • S020 records tagged evaluation: 272 count (multi-label)
  • S021 records tagged dataset: 222 count (multi-label)
  • S022 records tagged agentic: 105 count (multi-label)

What does today's evidence fail to show, and what would change the reading?

high confidence

The evidence does not establish a representative field-wide shift, independently verified superiority, or corroborated adoption. Category observation averages declined across the comparable recent and prior windows, but those are movements inside a keyword-filtered feed. The listed download changes are cumulative across each artifact’s stated tracked span and come from a single source, so they are not corroborated movement or one-day demand.

Today’s records show what the radar captured, not what the whole AI field adopted or proved effective. The packet lacks independent reproductions of the highlighted releases, matched evaluations on common tasks, and confirmation of download movement from another source. Public attention observations also indicate discussion rather than a release, update, technical result, or deployment.

Takeaway: The reading would strengthen with independent reproductions, common-task comparisons, per-metric confirmation from multiple sources, restored coverage from unavailable connectors, and sustained results across later comparable windows. Until then, use the declines as feed-level monitoring signals and the releases as evaluation candidates, not evidence of broad technical or market change.

Another reading: Because the recent and prior windows used identical measurement settings and cleared coverage requirements, the declines may reflect a genuine reduction in captured activity rather than collection noise. Some artifacts were also seen by multiple sources. However, independent artifact sightings do not confirm the same metric, and the download movements remain uncorroborated.

  • S002 public attention observations captured today: 15
  • S026 tracked artifacts today seen by more than one data source: 30
  • S027 downloads change for lmarena-ai/leaderboard-dataset: 182,705.0 downloads
  • S028 downloads change for open-llm-leaderboard/requests: 60,232.0 downloads
  • S029 downloads change for huggingface-projects/drlc-leaderboard-data: 39,135.0 downloads
  • S030 downloads change for vedangfake/chess-slm-benchmark: 23,218.0 downloads
  • S031 downloads change for hf-benchmarks/transformers: 16,434.0 downloads
  • S032 downloads change for alexshpunt/explicit-edit-benchmark: 15,442.0 downloads
  • S033 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 13,641.0 downloads
  • S034 downloads change for RyeAI/danish-asr-leaderboard: 6,710.0 downloads
  • S035 daily-average change in benchmark observations: -115.57 observations per day
  • S036 daily-average change in evaluation observations: -79.57 observations per day
  • S037 daily-average change in dataset observations: -43.14 observations per day
  • S038 daily-average change in agentic observations: -16.0 observations per day
  • S039 daily-average change in data_quality observations: -2.71 observations per day

Evidence sources

Where this came from, and what it does not cover

Built from the validated daily snapshot and its 573 evidence records.

Briefing model: gpt-5.6-sol.

It read 152 of 573 records.

These findings describe a keyword-filtered feed, not the AI field as a whole. Only 152 ranked evidence records were supplied from today’s 573-record corpus, so lower-ranked artifacts were not reviewed here. Brave, OpenReview, and Semantic Scholar were unavailable, and some attention collectors reported unread or stale records; no attention-based conclusion is included.