Benchmark Radar™
Star —

AgentJudgeBench and 4 more in AI benchmark news · Nov 20, 2026

626evidence observations
12sources represented
19public-attention signals

Featured on this day

  1. AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
  2. TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents
  3. Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation
  4. MACRO: A Benchmark for Clarification-Aware Multi-Agent Orchestration
  5. Do LLM-Generated Web Apps Behave Correctly? A Contract-Grounded Benchmark for Behavioral Conformance

Daily briefing

  1. The new AgentJudgeBench release tests whether large language models used as automated judges can grade tool-calling workflows represented as dependency graphs. Its 3,808 instances span six graph structures and three difficulty levels, and compare judging with and without ground-truth workflow information. This targets structured execution rather than open-ended text preferences. The separate scale-reversal probe in today’s feed also tests whether judges follow rubric meaning rather than numerical conventions. [E010, E060] Why it matters: If you use model-based graders for agent evaluations, ordinary agreement scores may not establish that the grader understands workflow dependencies or the rubric itself. These releases provide targeted checks that can inform whether to retain automated grading, supply execution ground truth, or add human review for complex workflows. [E010, E060] Evidence: E010, E060. Medium confidence.
  2. The new TraceProbe benchmark runs agents in executable sandboxes, records normalized action traces, and replays tool actions to locate where safety risks entered and propagated. It scores both final outcomes and hidden rule violations, rather than treating an apparently acceptable endpoint as sufficient evidence of safe execution. [E008] Why it matters: If you evaluate agents that use files, tools, memory, retrieved content, or other agents, TraceProbe offers a way to distinguish a safe-looking result from an unsafe path to that result. That can change whether a security suite is selected for release gating, incident diagnosis, or comparison of safeguards. [E008] Evidence: E008. Medium confidence.
  3. The new MESSIER corpus standardizes 957,611 evaluation records from 30 benchmarks, covering 745 agents, 11,891 tasks, and 74,263 verifiers, while adding runs on six underrepresented professional and scientific benchmarks. It maps otherwise heterogeneous tasks, agent scaffolds, scoring rules, and result records into a common schema. [E004] Why it matters: If you are choosing a suite for cross-benchmark agent comparison, MESSIER may reduce the need to rerun every system merely to obtain comparable records. Its common schema also makes verifier and scaffold differences visible, helping teams decide whether an apparent performance difference reflects the agent or the evaluation setup. [E004] Evidence: E004. Medium confidence.
  4. The new MACRO benchmark evaluates how multi-agent systems handle underspecified travel requests. Its 107 fully annotated tasks include 703 process milestones, expected user clarifications, task-state requirements, and standardized interfaces between specialized agents, so scoring can examine how a system reached its result rather than only whether the trip plan succeeded. [E005] Why it matters: If your product must ask users for missing constraints before coordinating dependent subtasks, end-to-end success can hide failures such as guessing preferences or losing state. MACRO provides process-level checkpoints that can influence which orchestration design, clarification policy, or agent interface you choose. [E005] Evidence: E005. Medium confidence.
  5. The new ConformWeb benchmark derives hidden interaction sequences from public behavioral contracts and executes them against generated web applications. Its protocol tests whether an application preserves specified behavior across dynamic workflows without requiring an undisclosed reference implementation. [E011] Why it matters: If you compare coding agents on web-application generation, screenshots or shallow functionality checks may reward plausible interfaces that fail under interaction. Contract-grounded hidden workflows offer a basis for selecting systems on behavioral conformance while keeping the expected public scope visible to the system being tested. [E011] Evidence: E011. Medium confidence.

What arrived

What benchmarks, datasets, or evaluation methods did the radar first see today?

medium confidenceNot enough evidence

Within this captured feed, first-seen releases covered multi-stage medical agents, crisis communication, cross-benchmark agent results, clarification-aware orchestration, automated retrieval-system development, execution safety, and contract-based web behavior. They also introduced proof-verified benchmark rewriting and a judge-scale reversal probe. A separately discovered record proposed a resampling-based test for comparing automated judges with human annotators.

Today’s first-seen material tests systems in more realistic workflows: changing clinical cases, public crises, underspecified requests, tool execution, and interactive software. Other arrivals focus on making evaluation itself more trustworthy by checking rewritten answers with formal proofs, exposing scoring-scale bias, or statistically comparing machine graders with people.

Takeaway: The registry confirms a substantial first-seen cohort, but the available evidence supports only a representative map rather than a complete inventory. These are first observations by this keyword-filtered radar, not proof that the artifacts are new to the field.

Another reading: The packet does not include full descriptions for every first-seen artifact, and some records were discoveries rather than releases. Collection changes and unavailable sources could also shape what appeared today, so the mix should not be interpreted as a field-wide shift.

  • S023 artifacts first observed by the radar today: 286

Which of today's arrivals document how they score an answer?

medium confidenceNot enough evidence

The clearest examples are TraceProbe, which scores rule violations from execution traces; RePro, which validates regenerated mathematical answers with formal proofs; a weight-editing benchmark that scores held-out success, side effects, and accuracy loss; and Sampling Config Benchmark, which uses unit tests for coding and repair tasks. A classification repository explicitly lists accuracy, precision, recall, error-balancing scores, confusion matrices, and operating-characteristic curves.

These arrivals do more than say whether an output looks good. They check observable behavior: whether actions break rules, whether a mathematical answer can be formally proved, whether edited models still work on unseen images, whether generated code passes tests, and whether predictions match labeled outcomes.

Takeaway: Several arrivals expose concrete grading mechanisms, especially executable tests, formal verification, trace-based rules, and held-out measurements. However, the summaries do not provide complete rubrics, aggregation rules, thresholds, or tie handling for every arrival, so this is a confirmed subset rather than an exhaustive answer.

Another reading: Some named mechanisms evaluate trajectories, model edits, distributions, or classifiers rather than a single free-text answer. The available summaries may also overstate documentation quality because the full scoring implementations and protocols were not inspected.

  • S023 artifacts first observed by the radar today: 286

What is still moving

Which artifacts the radar already tracked moved measurably, and over what span?

high confidenceNot enough evidence

The registry contains artifact-level download movements, but the evidence packet supplies no citable evidence records for those artifacts.

The listed changes cover each artifact’s full tracked span and are cumulative, not one-day changes. The renderer can show the artifact names, download changes, and spans from the cited statistics, but the required artifact evidence citations are missing.

Takeaway: Treat the listed statistics as provisional movement records within this keyword-filtered feed. A publishable artifact-by-artifact answer requires matching evidence IDs.

Another reading: The stat registry itself includes artifact names, spans, and source URLs, which may be operationally sufficient. However, the grounding rules separately require evidence IDs for claims about specific artifacts.

  • S026 downloads change for lmarena-ai/leaderboard-dataset: 322,942.0 downloads
  • S027 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 40,485.0 downloads
  • S028 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 30,469.0 downloads
  • S029 downloads change for Rapidata/text-to-image-human-feedback-benchmark: 9,117.0 downloads
  • S030 downloads change for lvkaokao/benchmark_data: 5,997.0 downloads
  • S031 downloads change for RyeAI/danish-asr-leaderboard: 5,030.0 downloads
  • S032 downloads change for INSAIT-Institute/LADDER-Evaluation-Episodes: 4,975.0 downloads
  • S033 downloads change for crawlcheck/agent-safety-benchmark: 4,759.0 downloads

Which of that movement is corroborated by more than one data source?

high confidenceNot enough evidence

None of the registry-listed movement statistics is marked as corroborated by more than one source.

Each listed download change comes from a single source. The separate multi-source sighting count does not establish that multiple sources measured the same movement.

Takeaway: Do not describe the listed download movements as corroborated. Identifying any other corroborated movements requires corresponding registry statistics and artifact evidence citations.

Another reading: The tracked-artifact payload flags some other movements as corroborated, but those entries lack corresponding movement statistics and citable evidence IDs, so they cannot be reported under the grounding rules.

  • S025 tracked artifacts today seen by more than one data source: 12
  • S026 downloads change for lmarena-ai/leaderboard-dataset: 322,942.0 downloads
  • S027 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 40,485.0 downloads
  • S028 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 30,469.0 downloads
  • S029 downloads change for Rapidata/text-to-image-human-feedback-benchmark: 9,117.0 downloads
  • S030 downloads change for lvkaokao/benchmark_data: 5,997.0 downloads
  • S031 downloads change for RyeAI/danish-asr-leaderboard: 5,030.0 downloads
  • S032 downloads change for INSAIT-Institute/LADDER-Evaluation-Episodes: 4,975.0 downloads
  • S033 downloads change for crawlcheck/agent-safety-benchmark: 4,759.0 downloads

What it means

What should someone building or evaluating AI systems do differently today?

medium confidence

Within this captured feed, several newly observed releases emphasize evaluating behavior across execution steps, hidden interactions, judge reliability, and training-data contamination rather than relying only on final answers or aggregate scores.

Review your evaluation plan for blind spots. Add tests that follow tool use and intermediate actions, verify that generated applications obey requirements during interaction, check automated graders against trusted answers, and screen benchmark material for possible training-data overlap. These practices are illustrated by specific releases, not yet validated as broadly superior methods.

Takeaway: Treat today’s records as a prompt to strengthen evaluation design, not as proof of a field-wide shift. Pilot the relevant methods on your own tasks, preserve traces and ground truth, and require reproducible evidence before changing deployment gates.

Another reading: The strongest competing reading is that these are mostly proposal-level, newly captured artifacts whose summaries reflect author claims. They may address narrow settings, and the feed provides no independent replication showing that adopting them improves real production decisions.

  • S003 records with event kind released: 301
  • S018 records tagged benchmark: 436 count (multi-label)
  • S019 records tagged evaluation: 303 count (multi-label)
  • S021 records tagged agentic: 87 count (multi-label)
  • S023 artifacts first observed by the radar today: 286

What does today's evidence fail to show, and what would change the reading?

high confidenceNot enough evidence

The feed does not establish a time trend, broad field direction, benchmark quality, or independently corroborated adoption. The comparison window is uncertified, source coverage is incomplete, and the reported download movements come from single sources across each artifact’s full tracked span.

Differences from other days could reflect collection changes rather than changes in AI work. Download movement shows accumulated activity over varying tracked spans, not daily momentum or verified use. Release descriptions also do not establish that datasets are clean, tests are valid, or reported methods work outside their chosen settings.

Takeaway: The reading would change with a certified like-for-like history, restored source coverage, independent reports of the same movements, accessible result tables, reproducible code and data, contamination audits, and external replications on representative production tasks. Until then, conclusions should remain scoped to this keyword-filtered feed.

Another reading: A competing reading is that the breadth of newly captured benchmark and evaluation records is still useful for discovery despite weak comparability. However, discovery volume alone cannot establish momentum, quality, adoption, or practical impact.

  • S001 evidence records captured today: 626
  • S023 artifacts first observed by the radar today: 286
  • S025 tracked artifacts today seen by more than one data source: 12
  • S026 downloads change for lmarena-ai/leaderboard-dataset: 322,942.0 downloads
  • S027 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 40,485.0 downloads
  • S028 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 30,469.0 downloads
  • S029 downloads change for Rapidata/text-to-image-human-feedback-benchmark: 9,117.0 downloads
  • S030 downloads change for lvkaokao/benchmark_data: 5,997.0 downloads
  • S031 downloads change for RyeAI/danish-asr-leaderboard: 5,030.0 downloads
  • S032 downloads change for INSAIT-Institute/LADDER-Evaluation-Episodes: 4,975.0 downloads
  • S033 downloads change for crawlcheck/agent-safety-benchmark: 4,759.0 downloads
  • S034 daily-average change in benchmark observations: -33.86 observations per day
  • S035 daily-average change in dataset observations: -16.86 observations per day
  • S036 daily-average change in agentic observations: -2.71 observations per day
  • S037 daily-average change in evaluation observations: 1.43 observations per day
  • S038 daily-average change in data_quality observations: 0.57 observations per day

Evidence sources

Where this came from, and what it does not cover

Built from the validated daily snapshot and its 626 evidence records.

Briefing model: gpt-5.6-sol.

It read 165 of 626 records.

This is a keyword-filtered, non-representative feed. Only 165 of 626 corpus evidence records were supplied for analysis, although none of those selected records were dropped for size. Brave, OpenReview, and Semantic Scholar were unavailable, and several attention collectors had partial reads or stale carry-forwards. The findings describe source-authored releases in the captured evidence, not independently validated benchmark quality or field-wide prevalence.