Benchmark Radar™
Star —

PrincipalBench v2 and 8 more in AI benchmark news · Nov 13, 2026

500evidence observations
11sources represented
11public-attention signals

Featured on this day

  1. PrincipalBench v2: A Validity-First Benchmark for LLM Orchestrators in Multi-Agent Pipelines
  2. Journeyman: a process-quality benchmark for LLM agents
  3. Headroom Decision Procedure Examples: dataset for "Headroom Before Protocol Effects: A Decision Procedure for Human-Agent Evaluation"
  4. Trust-Calibration Benchmark (TCB): dataset for "Qualification by Calibration: A Readable Benchmark for Admitting Language Models to Human-Computation Tasks"
  5. ReconRAG: Ingest-Time Version Reconciliation for Retrieval-Augmented Generation over Evolving Regulations (Code and Benchmark Dataset)
  6. Automated Biomedical Waste Segregation Using Deep Learning: A Duplicate-Aware Classification Benchmark and Disposal-Safety Assessment
  7. twill-iclr2027/twine-benchmark
  8. Agent Decomposition Granularity Benchmark (ADGB)
  9. Headroom Decision Procedure Examples: dataset for "Headroom Before Protocol Effects: A Decision Procedure for Human-Agent Evaluation"

Daily briefing

  1. PrincipalBench v2 is a new report that audits its predecessor against the underlying code and data. It finds that 433 of 500 v1 tasks were placeholders without real questions or gold answers, every episode included a failure—letting an always-flag policy score perfectly—and only three distinct failures were tested. The report says v1 cannot support model comparisons. (E030) Why it matters: If you are choosing a model to coordinate worker agents, v1 scores should not drive that choice. The release demonstrates why benchmark admission checks must inspect task substance, failure prevalence, and trivial policies before comparing orchestrators. Evidence: E030. High confidence.
  2. Journeyman is a new benchmark that evaluates how an agent completes work rather than only whether it passes. Across eight seed-rebuilt simulated jobs, it scores ten process dimensions, including repeated futile actions, stopping behavior, acknowledgment of unreachable goals, and repetition of planted false information; simulation state can expose claimed actions that never occurred. (E017) Why it matters: If you are evaluating operational agents, Journeyman lets you distinguish systems with identical final outcomes but different efficiency, honesty, and failure handling. That can change product selection when the path to completion carries cost or risk. Evidence: E017. High confidence.
  3. The new Trust-Calibration Benchmark packages 60 audited yes-or-no items across five domains, responses from nine language models under three administration protocols, repeated sampling, confidence-elicitation experiments, transfer-study data, and offline analyzers backed by a 5,700-response cache. (E005) Why it matters: If you are deciding whether a language model may perform human-computation tasks, this artifact supports testing whether expressed confidence remains useful across protocols and transfers—not merely whether average answers are correct. Its cached responses and analyzers also make reported results independently reproducible. Evidence: E005. High confidence.
  4. The new TWINE benchmark evaluates proactive robotic agents with 360 cases arranged as 180 controlled pairs across 60 held-out kitchen-scene families. It also provides an independent generalization set of 156 cases across 69 newly decoded scene families, together with images, reference obligations, and scoring rubrics. (E007) Why it matters: If you are selecting a benchmark for proactive embodied agents, the paired cases can isolate the effect of controlled scenario changes, while the separate scene families test transfer beyond the main set. This is more informative than relying on one aggregate score over related scenes. Evidence: E007. High confidence.
  5. ReconRAG is a new code-and-benchmark release for retrieval-augmented generation, where a model answers using retrieved documents, over regulations that change over time. It covers 57 Indonesian regulation pairs, with official amendment instructions supplying gold labels for 47 pairs and three language models supplying weaker silver labels for ten full replacements; it also includes validated questions, frozen prompts, graded answers, and reproduction code. (E012) Why it matters: If you are evaluating regulatory question-answering systems, this benchmark tests whether retrieval reconciles document versions rather than mixing current and superseded rules. Model comparisons should keep the official gold subset separate from the machine-labeled silver subset because their evidence quality differs. Evidence: E012. High confidence.
  6. The new Agent Decomposition Granularity Benchmark tests 512 clinical forms while splitting extraction into one, two, four, or eight model calls. Independent fields changed little, but chained billing fields degraded sharply; one model fell from 1.000 to 0.523–0.641 on English notes, with most errors tied to an ambiguous fee description. (E032) Why it matters: If you are designing a multi-call agent pipeline, this benchmark shows that call count alone is not the useful comparison. Evaluation needs to separate independent from dependent fields and inspect prompt ambiguity, because decomposition can change which interpretation propagates through later steps. Evidence: E032. High confidence.
  7. Headroom Decision Procedure Examples is a new worked evaluation artifact for human-agent team protocols. It packages result tables and generating scripts for checking a protocol against a matched no-protocol baseline, determining whether meaningful improvement remains possible, and mapping paired uncertainty to an operational action. (E006) Why it matters: If you are assessing whether a collaboration protocol helps, this procedure prevents attributing value to the protocol when the baseline is unmatched or already leaves little room to improve. It changes the evaluation decision from reporting a score difference to deciding whether that difference is interpretable and actionable. Evidence: E006. Medium confidence.

What arrived

What benchmarks, datasets, or evaluation methods did the radar first see today?

high confidence

This captured feed first observed a substantial group of artifacts today. Notable releases included benchmarks for federated governance, radiotherapy dose calculation, trust calibration, proactive agents, regulatory retrieval, and agent process quality [E003, E004, E005, E007, E012, E017].

The new-to-radar material covered governance workflows, medical dose prediction, model confidence, household-agent behavior, changing regulations, soil moisture, invoice extraction, and refrigeration calculations [E003, E004, E005, E007, E012, E023, E038, E039]. Evaluation ideas included multiple AI judges, a headroom decision procedure, robustness certification, and duplicate-aware testing [E001, E006, E008, E025].

Takeaway: Treat these as first sightings within this keyword-filtered radar, not a representative catalog or evidence that every item was newly created today. The arrivals mix releases with previously published material that the radar merely discovered, and the category tags overlap.

Another reading: A first radar sighting is not necessarily a new field release. For example, the feed marked the mathematical supervision corpus as discovered even though its stated publication preceded today [E115]. The selected evidence also cannot support an exhaustive qualitative review of every first-observed artifact.

  • S022 artifacts first observed by the radar today: 295

Which of today's arrivals document how they score an answer?

high confidence

The clearest first-seen releases with explicit scoring information were the trust-calibration benchmark, TWINE, Journeyman, the code-graph benchmark pack, the invoice-extraction benchmark, and the refrigeration answer-verification benchmark [E005, E007, E017, E026, E038, E039].

Their documentation describes gold answers and reproducible analyzers, reference obligations and rubrics, process-quality dimensions, independent rescoring from saved predictions, field-by-field comparison, or reference results with tolerances and unit-conversion rules [E005, E007, E017, E026, E038, E039]. These are stronger scoring disclosures than merely publishing model outputs or saying answers were graded.

Takeaway: For teams seeking inspectable evaluation, the invoice and refrigeration releases provide direct answer-checking rules, while trust calibration and the code-graph release emphasize reproducibility [E005, E026, E038, E039]. TWINE and Journeyman document rubric-based scoring for agent behavior rather than only final-answer correctness [E007, E017].

Another reading: The evidence summaries do not establish that every rubric, formula, weighting rule, or edge case is fully specified. ReconRAG, for example, reports graded model answers but its summary does not explain the grading procedure [E012]. Full artifact inspection would be required before calling any scorer completely reproducible.

  • S022 artifacts first observed by the radar today: 295

What is still moving

Which artifacts the radar already tracked moved measurably, and over what span?

low confidenceNot enough evidence

The registry supplies artifact-level download movement for the entries referenced in the statistics. Each value is cumulative across its listed first-observation-to-today span, rather than a one-day change. Artifact-level evidence records were not supplied.

Read each listed movement as the total download-count change between the artifact’s first and latest tracked observations. The relevant artifact names, values, and spans appear in the referenced statistics.

Takeaway: The registered entries show measurable cumulative download changes within this keyword-filtered feed. They are attention measurements, not evidence of new releases, updates, or broader field movement.

Another reading: The broader tracked-artifact table contains additional metric changes without corresponding registry statistics, so the registered set may not be exhaustive. Missing artifact-level evidence also prevents independent verification.

  • S025 downloads change for lmarena-ai/leaderboard-dataset: 287,518.0 downloads
  • S026 downloads change for open-llm-leaderboard/requests: 64,956.0 downloads
  • S027 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 38,863.0 downloads
  • S028 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 32,794.0 downloads
  • S029 downloads change for lukakuma/web-trajectories-preference-raw: 10,855.0 downloads
  • S030 downloads change for prvsiyan's Extended Name That LLM Dataset: 10,472.0 downloads
  • S031 downloads change for qimma/leaderboard-details: 7,948.0 downloads
  • S032 downloads change for RyeAI/danish-asr-leaderboard: 7,470.0 downloads

Which of that movement is corroborated by more than one data source?

low confidenceNot enough evidence

None of the registered download movements is corroborated by more than one source. The aggregate multisource-sighting statistic does not establish that separate sources measured the same metric.

An artifact appearing in several sources is different from several sources independently confirming its download change. The supplied movement statistics are each marked single-source.

Takeaway: Treat the registered download changes as uncorroborated. Naming any additional corroborated movement requires a registry statistic and artifact-level evidence showing that separate sources measured the same metric.

Another reading: The tracked metadata includes a repository entry marked corroborated across separate source labels. However, it lacks a movement statistic and supporting evidence IDs, so it cannot be reported as verified under these grounding rules.

  • S024 tracked artifacts today seen by more than one data source: 9
  • S025 downloads change for lmarena-ai/leaderboard-dataset: 287,518.0 downloads
  • S026 downloads change for open-llm-leaderboard/requests: 64,956.0 downloads
  • S027 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 38,863.0 downloads
  • S028 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 32,794.0 downloads
  • S029 downloads change for lukakuma/web-trajectories-preference-raw: 10,855.0 downloads
  • S030 downloads change for prvsiyan's Extended Name That LLM Dataset: 10,472.0 downloads
  • S031 downloads change for qimma/leaderboard-details: 7,948.0 downloads
  • S032 downloads change for RyeAI/danish-asr-leaderboard: 7,470.0 downloads

What it means

What should someone building or evaluating AI systems do differently today?

medium confidence

Use today’s captured releases to strengthen evaluation design, not to chase a supposed field-wide shift. Add matched baselines, check whether the task leaves meaningful room for improvement, audit labels, preserve evaluation logs, and score how agents work as well as whether they finish.

Several captured releases provide concrete patterns: matched no-protocol comparisons and headroom checks, audited labels with reproducible analysis, process-focused agent grading, and saved held-out evaluation records. These are examples to inspect and adapt, not proof that any benchmark is valid for your system.

Takeaway: Before accepting a model or agent improvement, require a relevant baseline, uncertainty reporting, reproducible records, and tests that reflect the intended deployment. Treat releases, updates, and public attention as different signals within this keyword-filtered feed.

Another reading: The packet mostly supplies repository metadata and author descriptions rather than independent validation. Several captured agent-benchmark releases have little or no summary detail, so adopting their methods without inspecting code, data, licenses, and leakage risks could add ceremony without improving decisions.

  • S003 records with event kind updated: 230
  • S004 records with event kind released: 224
  • S017 records tagged benchmark: 336 count (multi-label)
  • S018 records tagged dataset: 230 count (multi-label)
  • S019 records tagged evaluation: 206 count (multi-label)
  • S020 records tagged agentic: 65 count (multi-label)
  • S022 artifacts first observed by the radar today: 295

What does today's evidence fail to show, and what would change the reading?

high confidenceNot enough evidence

Today’s packet does not establish a field-wide trend, benchmark quality, real-world effectiveness, or broad adoption. The registry has no certified comparison window, and the reported download movements come from single sources across each artifact’s full tracked span rather than representing daily change.

Differences from nearby days may reflect collection changes because source coverage and comparability are incomplete. Release metadata also cannot show whether datasets are clean, benchmarks resist contamination, scores reproduce independently, or systems transfer to production conditions.

Takeaway: The reading would change with a certified like-for-like history, restored source coverage, independent replication, shared evaluations across competing systems, documented dataset audits, and movement confirmed by more than one source. Until then, interpret this as a discovery feed rather than a market or research census.

Another reading: Some releases include unusually concrete reproducibility material, such as label audits, cached responses, frozen prompts, executable analysis, and preserved evaluation logs. That makes selected artifacts potentially decision-useful even though the packet cannot support broader claims about the field.

  • S024 tracked artifacts today seen by more than one data source: 9
  • S025 downloads change for lmarena-ai/leaderboard-dataset: 287,518.0 downloads
  • S026 downloads change for open-llm-leaderboard/requests: 64,956.0 downloads
  • S027 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 38,863.0 downloads
  • S028 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 32,794.0 downloads
  • S029 downloads change for lukakuma/web-trajectories-preference-raw: 10,855.0 downloads
  • S030 downloads change for prvsiyan's Extended Name That LLM Dataset: 10,472.0 downloads
  • S031 downloads change for qimma/leaderboard-details: 7,948.0 downloads
  • S032 downloads change for RyeAI/danish-asr-leaderboard: 7,470.0 downloads
  • S033 daily-average change in benchmark observations: 26.86 observations per day
  • S034 daily-average change in agentic observations: -10.57 observations per day
  • S035 daily-average change in evaluation observations: -10.43 observations per day
  • S036 daily-average change in dataset observations: 2.86 observations per day
  • S037 daily-average change in data_quality observations: 0.29 observations per day

Evidence sources

Where this came from, and what it does not cover

Built from the validated daily snapshot and its 500 evidence records.

Briefing model: gpt-5.6-sol.

It read 131 of 500 records.

This is a keyword-filtered, non-representative feed. Only 131 of 500 corpus evidence records were supplied for analysis, although no selected records were dropped for size. Brave, OpenReview, and Semantic Scholar were unavailable, and several attention collectors returned partial or stale measurements. Comparable history was insufficient to infer category-share movement, so these findings describe individual releases in the captured evidence rather than field-wide trends.