Benchmark Radar™
Star —

agent-benchmarks-clean and 4 more in AI benchmark news · Nov 19, 2026

599evidence observations
13sources represented
19public-attention signals

Featured on this day

  1. zhiyuanhucs/agent-benchmarks-clean
  2. PIPETTE: Prompt Injection Purple-Teaming for Robust LLM Agents
  3. Benchmarking LLM abilities on postdoc-level natural science problems
  4. VideoACE: Benchmarking Agentic Video Understanding through High-Frame-Rate Actions
  5. Homology Leakage Inflates Performance on a Protein Functional-Class Benchmark

Daily briefing

  1. The new agent-benchmarks-clean release packages coding-agent tasks as immutable Linux containers with pinned configurations and SHA-256-verified layers. Its companion evaluation package fixes task subsets, harness versions, and scoring methods across Terminal Bench, software-engineering, and code-review evaluations. This is a new reproducibility artifact rather than a leaderboard update. [E001, E036] Why it matters: If you compare coding agents across machines or over time, this package lets you hold the execution environment and evaluator constant. That makes it easier to attribute score changes to the agent instead of changed dependencies, container contents, or harness versions. [E001, E036] Evidence: E001, E036. High confidence.
  2. The new PIPETTE release evaluates prompt-injection attacks by whether they produce useful defenses, not only by attack success rate. Its source-authored description reports that a high attack success rate during automated red-teaming does not necessarily translate into stronger agents after adversarial training. [E009] Why it matters: If you are selecting an automated red-teaming method for tool-using agents, attack success alone is an incomplete selection metric. PIPETTE makes downstream robustness gains a separate evaluation target, connecting offensive testing to the defensive outcome the test is meant to improve. [E009] Evidence: E009. High confidence.
  3. The new STEMBench release scores 12 postdoctoral-level problems in high-energy physics, astrophysics, and neuroscience using each field’s own measure of solution quality. Every problem also includes a no-domain-knowledge anchor and an expert anchor, rather than relying on answer matching or generic judging. [E017] Why it matters: If you are evaluating research agents, this design tests whether their work improves the quantity scientists actually optimize, not merely whether they reproduce a known answer. The anchors also provide interpretable reference points when raw scores differ across scientific fields. [E017] Evidence: E017. High confidence.
  4. The new VideoACE benchmark tests whether a video agent uses interaction opportunities to improve understanding. It contains 1,000 questions over high-frame-rate action videos, where brief changes reward targeted observation, and supplies temporal evidence annotations and screened user hints. [E010] Why it matters: If you are choosing a suite for interactive video systems, VideoACE separates active evidence gathering from passive video question answering. That distinction matters for products whose models can seek additional frames or revisit specific moments before answering. [E010] Evidence: E010. High confidence.
  5. Two new releases in the captured feed independently audit train-test leakage in scientific image and sequence benchmarks. The protein study replaces random splits with identity-controlled splits that keep related sequences together, while the BRISC brain-tumor audit reports exact and near-duplicate images crossing the official split. [E052, E077] Why it matters: If you evaluate models on biological sequences or medical images, random record-level splits may measure recognition of related or duplicated examples rather than generalization. These releases support making relationship-aware splitting and duplicate checks part of benchmark acceptance before comparing model rankings. [E052, E077] Evidence: E052, E077. High confidence.

What arrived

What benchmarks, datasets, or evaluation methods did the radar first see today?

high confidence

In this captured feed, first-seen releases included Agent Recovery Bench, VideoACE, RoughCut-Bench, ShadowTransfer, RFBench, MARBLE, and RevertBench. The radar also first discovered FreshSearch, a multi-agent coordination benchmark, FVSCORE, and DrawScore today.

The arrivals cover tool-using agent recovery, interactive video understanding, video editing, geographic transfer, radio-frequency design, spatial planning, evidence revision, search freshness, agent coordination, generated-video faithfulness, and engineering-drawing checks. These are distinct releases or discoveries, not evidence of a field-wide trend.

Takeaway: The radar first observed many artifacts today, with representative additions spanning agent behavior, multimodal tasks, scientific design, robustness, and specialized scoring methods. “First observed” means new to this keyword-filtered feed, not necessarily newly created or newly released.

Another reading: Some first-seen records are not genuinely new artifacts: the MetaGameBO dataset explicitly describes itself as a mirror containing no new data. Collection gaps and unavailable sources also mean this list cannot establish completeness across the field.

  • S024 artifacts first observed by the radar today: 288

Which of today's arrivals document how they score an answer?

high confidence

The clearest examples are STEMBench, RevertBench, the decision-model order-invariance benchmark, the agent benchmark evaluation package, the Arabic OCR prediction release, and FVSCORE. Each names a scoring rule, oracle, reference structure, metric pipeline, or evaluation code.

STEMBench uses each scientific field’s own measure and comparison anchors. RevertBench uses a deterministic Boolean oracle. The order-invariance dataset supplies explicit decision rules and reference answers. The agent package names success, resolution, precision, recall, and their combined measure. The OCR release provides references, predictions, and scoring code. FVSCORE checks explicit requirements over time.

Takeaway: These arrivals expose how outputs become scores through field-specific measures, deterministic truth rules, reference answers, executable evaluation code, or proposition-level checks. That makes their scoring approach identifiable from the captured documentation, although the packet does not verify every implementation.

Another reading: The evidence consists mainly of titles and summaries rather than full technical documentation. Movie2Game-Bench, for example, describes several assessment components but the captured summary does not specify how they are combined, so other arrivals may document scoring more fully than this packet reveals.

What is still moving

Which artifacts the radar already tracked moved measurably, and over what span?

low confidenceNot enough evidence

Artifact-level identification is not adequately grounded from the supplied materials.

The registry contains cumulative download movements measured across each artifact’s full tracked span, rather than one-day changes. However, no E-coded evidence accompanies those artifact records, so the specific names and spans cannot be reported under the evidence rules.

Takeaway: Obtain E-coded source records before presenting the affected artifacts and their full measurement spans. This limitation applies to this keyword-filtered captured feed only.

Another reading: The statistic labels themselves contain artifact names and measurement windows, so they provide a usable registry-level answer. They still do not satisfy the separate requirement for E-coded evidence on claims about specific artifacts.

  • S027 downloads change for lmarena-ai/leaderboard-dataset: 315,672.0 downloads
  • S028 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 40,165.0 downloads
  • S029 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 30,777.0 downloads
  • S030 downloads change for Rapidata/text-to-image-human-feedback-benchmark: 8,593.0 downloads
  • S031 downloads change for lvkaokao/benchmark_data: 5,677.0 downloads
  • S032 downloads change for RyeAI/danish-asr-leaderboard: 4,998.0 downloads
  • S033 downloads change for crawlcheck/agent-safety-benchmark: 4,673.0 downloads
  • S034 downloads change for RoboDyna/robodyna-benchmark-v2: 4,642.0 downloads

Which of that movement is corroborated by more than one data source?

high confidence

None of the registered movement is corroborated by more than one data source.

Every registered download movement came only from Hugging Face. The changes cover each artifact’s whole tracked span and are not one-day movements.

Takeaway: Treat all registered movement as single-source measurement within this keyword-filtered captured feed, not as independently confirmed movement across the field.

Another reading: The feed captured multiple-source sightings for some tracked artifacts, but those sightings did not independently measure the same movement metric and therefore do not constitute corroboration.

  • S026 tracked artifacts today seen by more than one data source: 8
  • S027 downloads change for lmarena-ai/leaderboard-dataset: 315,672.0 downloads
  • S028 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 40,165.0 downloads
  • S029 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 30,777.0 downloads
  • S030 downloads change for Rapidata/text-to-image-human-feedback-benchmark: 8,593.0 downloads
  • S031 downloads change for lvkaokao/benchmark_data: 5,677.0 downloads
  • S032 downloads change for RyeAI/danish-asr-leaderboard: 4,998.0 downloads
  • S033 downloads change for crawlcheck/agent-safety-benchmark: 4,673.0 downloads
  • S034 downloads change for RoboDyna/robodyna-benchmark-v2: 4,642.0 downloads

What it means

What should someone building or evaluating AI systems do differently today?

medium confidence

Add failure-oriented, task-specific regression tests, and do not change production systems solely because of leaderboard or download attention.

Today’s captured new releases cover recovery from tool failures, indirect prompt injection, active evidence gathering in video, and reproducible agent environments (E003, E009, E010, E001). These are useful prompts for testing blind spots, not proof that any benchmark is valid for your application.

Takeaway: Agent builders should test recovery after failed steps and attacks delivered through external data. Multimodal builders should test whether interaction actually improves answers. Pin environments and data versions, then rerun relevant tests on your own workload before adopting conclusions from this feed (E001, E003, E009, E010).

Another reading: This keyword-filtered feed is not representative, and the selected releases have not been independently validated here. Some benchmarks may have little operational relevance; one captured simulation benchmark explicitly reports no real-user testing or outcome evaluation (E015).

  • S003 records with event kind released: 277
  • S019 records tagged benchmark: 414 count (multi-label)
  • S020 records tagged evaluation: 271 count (multi-label)
  • S022 records tagged agentic: 75 count (multi-label)

What does today's evidence fail to show, and what would change the reading?

high confidenceNot enough evidence

Today’s feed does not establish a field-wide trend, quality ranking, adoption shift, or causal explanation.

There is no certified comparison window, category tags overlap, and differences between days may reflect collection changes. Cross-source sightings are sparse. The listed download movements are single-source cumulative changes over each artifact’s full tracked span, not daily movements or corroborated adoption measures.

Takeaway: A stronger reading requires comparable history under the same taxonomy and coverage, restoration of missing connectors, repeated independent measurement of the same metrics, and external reproduction of benchmark claims. Until then, use today’s material for discovery rather than trend or adoption conclusions.

Another reading: A competing reading is that the feed remains useful as a broad discovery index: it documents many captured records and concrete new releases even though it cannot establish wider direction. That supports investigation, but not generalization beyond the feed.

  • S001 evidence records captured today: 599
  • S019 records tagged benchmark: 414 count (multi-label)
  • S020 records tagged evaluation: 271 count (multi-label)
  • S021 records tagged dataset: 262 count (multi-label)
  • S026 tracked artifacts today seen by more than one data source: 8
  • S027 downloads change for lmarena-ai/leaderboard-dataset: 315,672.0 downloads
  • S028 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 40,165.0 downloads
  • S029 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 30,777.0 downloads
  • S030 downloads change for Rapidata/text-to-image-human-feedback-benchmark: 8,593.0 downloads
  • S031 downloads change for lvkaokao/benchmark_data: 5,677.0 downloads
  • S032 downloads change for RyeAI/danish-asr-leaderboard: 4,998.0 downloads
  • S033 downloads change for crawlcheck/agent-safety-benchmark: 4,673.0 downloads
  • S034 downloads change for RoboDyna/robodyna-benchmark-v2: 4,642.0 downloads
  • S035 daily-average change in evaluation observations: 25.29 observations per day
  • S036 daily-average change in benchmark observations: -9.0 observations per day
  • S037 daily-average change in agentic observations: 3.57 observations per day
  • S038 daily-average change in data_quality observations: 0.86 observations per day
  • S039 daily-average change in dataset observations: -0.43 observations per day

Evidence sources

Where this came from, and what it does not cover

Built from the validated daily snapshot and its 599 evidence records.

Briefing model: gpt-5.6-sol.

It read 169 of 599 records.

This briefing examined 169 selected evidence records from a 599-record keyword-filtered corpus, so it does not represent a full reading of the captured feed or the wider field. Brave, OpenReview, and Semantic Scholar were unavailable, and several attention collectors returned partial or stale measurements. Comparable history was also insufficient for category-share claims.