Benchmark Radar™
Star —

ForecastGym and 5 more in AI benchmark news · Nov 18, 2026

747evidence observations
14sources represented
13public-attention signals

Featured on this day

  1. ForecastGym: A Live, Controlled, and Replayable Benchmark for Forecasting Agents
  2. IH-Benchmark: A Conflict-Centered Benchmark for Instruction-Hierarchy Robustness in Large Language Models Benchmark page
  3. Verifying Agents in Rubric-Graded Environments
  4. RISE Humanities Data Benchmark Dataset: Complete Stored Results
  5. Where Does Hindsight Enter Clinical Agent Benchmarks? An Audit of Simulators and Case Construction
  6. SOTA or Luck? The Winner's Curse in LLM Leaderboards

Daily briefing

  1. ForecastGym is a new benchmark that records how evidence from four commercial web-search Application Programming Interfaces changes before forecast resolution, enabling live but replayable tests of forecasting agents with fixed evidence histories. Why it matters: If you are comparing forecasting models, search providers, or memory strategies, this design separates reasoning quality from differences in retrieved information and avoids retrospective searches that may expose later information. Evidence: E004. High confidence.
  2. IH-Benchmark is a new test suite for conflicting instructions across both system-versus-user and user-versus-tool priority levels, organized into seven categories of higher-priority constraints. Why it matters: If you are evaluating a tool-using assistant, this adds coverage that ordinary prompt-conflict tests miss: whether the model preserves authorized user instructions when tool output contains competing directions. Evidence: E002. High confidence.
  3. BankerVerifierBench is a new meta-evaluation dataset for the systems that grade agents, containing 2,464 human-judged rubric criteria from 24 investment-banking tasks and a nine-capability framework for diagnosing verifier behavior. Why it matters: If agent performance depends on an automated grader, this provides a way to test the grader itself rather than treating its verdict as ground truth, reducing the risk of selecting agents based on verifier errors. Evidence: E008. High confidence.
  4. The RISE Humanities Data Benchmark released its complete stored results as joinable tables: 3,861 runs, 168,916 requests, nearly four million metric observations, and per-request cost and runtime for historical-document tasks. Why it matters: If you are choosing models for humanities workflows, the release supports reanalysis across transcription, interpretation, structured extraction, quality, latency, and cost instead of relying only on a summarized leaderboard. Evidence: E001. High confidence.
  5. Benchmark leakage is a recurring design pressure in this feed. HindAudit separates three ways that knowledge of a final diagnosis can enter retrospective clinical-agent tests, while a respiratory-sound benchmark shows how patient overlap between training and test data can inflate reported performance. Why it matters: If you are assessing clinical systems, these artifacts make the data split and case-construction process part of the evaluation decision; a high score is less informative when the same patient or outcome-derived information can appear on both sides of the test. Evidence: E044, E086. High confidence.
  6. The new study “SOTA or Luck?” formalizes rank-induced bias in leaderboards, where selecting the highest noisy score rewards sampling luck, and proposes a bootstrap correction under stated assumptions about the model pool and benchmark tasks. Why it matters: If you are selecting a model from closely spaced leaderboard results, this means the displayed winner may not be the best underlying system; corrected estimates or uncertainty should influence procurement and research comparisons. Evidence: E054. Medium confidence.

What arrived

What benchmarks, datasets, or evaluation methods did the radar first see today?

high confidence

The captured feed first observed work on humanities-document tasks, instruction-conflict robustness, controlled forecasting agents, legal analogies, social-signal recognition, scientific workflows, and enterprise agents. Examples include the RISE benchmark, IH-Benchmark, ForecastGym, LARQS, Social Signal Bench, SciWorkBench, OfficePrism, and ATLAS-Finance. The last two were discovery records; the others were recorded as releases.

Today’s newly observed artifacts cover models reading historical documents, following higher-priority instructions, forecasting from controlled web evidence, understanding legal word relationships and social behavior, completing scientific work, and operating in simulated workplaces. These are examples from the keyword-filtered feed, not a complete or representative map of AI evaluation.

Takeaway: The radar first observed a large cohort of artifacts today, with several notable evaluations moving beyond short question answering toward workflows, tool use, evidence control, and specialized professional tasks. Use the cohort as a discovery list rather than evidence that these areas are becoming more common, because today lacks a certified comparison window.

Another reading: First observed by the radar does not mean newly created or released to the field. OfficePrism and ATLAS-Finance were discovery records, while the Litmiq record describes an evaluation run and release predating today’s capture. Collection timing and source coverage can therefore make older work appear among today’s arrivals.

  • S025 artifacts first observed by the radar today: 423

Which of today's arrivals document how they score an answer?

medium confidenceNot enough evidence

Clear examples include RISE, which scores against expert ground truth; the bladder-pathology challenge, which combines text overlap, keyword, semantic, and expert ratings; The Shared Word, which is deterministically scored; BankerVerifierBench, which uses human-judged rubric criteria; Think Before You Code, which combines several comparison metrics; GauntletScore, which applies a fixed agreement rule; CrisisBench, which uses an expert-calibrated judge; and Journeyman, which grades process dimensions.

These arrivals explain at least the basic path from an output to a result: compare it with an expert answer, apply a fixed answer checker, evaluate several qualities, require agreement between independent readers, or grade how an agent worked. Other records also mention graders or answer keys, including the coding-agent task collection and the ALE Science Benchmark.

Takeaway: The strongest scoring documentation in the captured summaries uses explicit references, deterministic checks, published metric combinations, calibrated judges, or auditable decision rules. The registry does not provide a count of arrivals meeting that standard, so the evidence supports named examples rather than a complete total.

Another reading: The summaries are not full method sections, and documentation depth varies. IH-Benchmark’s evaluator description is truncated, while Social Signal Bench criticizes a conventional score without fully exposing its replacement in the supplied text. A complete classification would require the full artifacts and untruncated scoring specifications.

  • S025 artifacts first observed by the radar today: 423

What is still moving

Which artifacts the radar already tracked moved measurably, and over what span?

high confidenceNot enough evidence

The registry contains cumulative download-movement entries for previously tracked artifacts, each covering its full first-to-last observation window rather than a one-day change. However, no E-tagged source records connect those entries to specific artifacts.

The captured feed measured changes across several tracking periods, but the required source citations are absent. Under the grounding rules, the individual artifacts and their movements therefore cannot be reported as substantiated.

Takeaway: Do not publish an artifact-level movement list until the underlying source records receive E identifiers. The captured feed is keyword-filtered and is not representative of the field.

Another reading: The registry labels, URLs, and window fields could be treated as sufficient structured evidence. However, doing so would bypass the explicit requirement for E-tagged support for claims about specific artifacts.

  • S028 downloads change for lmarena-ai/leaderboard-dataset: 307,398.0 downloads
  • S029 downloads change for open-llm-leaderboard/requests: -50,181.0 downloads
  • S030 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 39,917.0 downloads
  • S031 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 30,885.0 downloads
  • S032 downloads change for lukakuma/web-trajectories-preference-raw: 26,200.0 downloads
  • S033 downloads change for alexshpunt/explicit-edit-benchmark: 13,667.0 downloads
  • S034 downloads change for RyeAI/danish-asr-leaderboard: 5,194.0 downloads
  • S035 downloads change for lvkaokao/benchmark_data: 5,034.0 downloads

Which of that movement is corroborated by more than one data source?

high confidenceNot enough evidence

No corroborated movement can be reported under the grounding rules. The registry records multi-source sightings of tracked artifacts, but explicitly says that this does not establish that multiple sources measured the same metric.

Seeing an artifact in several places is different from having several places confirm the same change. The packet includes corroboration flags, but lacks corresponding movement statistics and E-tagged source records needed to substantiate them.

Takeaway: Treat all publishable movement statistics here as uncorroborated until per-metric, multi-source evidence is added. This conclusion applies only to the captured radar feed.

Another reading: The tracked-artifact packet marks some records as corroborated and lists multiple metric sources. That is a competing reading, but it cannot be cited or quantified because matching registry statistics and E identifiers are missing.

  • S027 tracked artifacts today seen by more than one data source: 29

What it means

What should someone building or evaluating AI systems do differently today?

medium confidence

Within this captured feed, newly released evaluation artifacts highlight practical checks for instruction-priority conflicts, replayable retrieval, verifier reliability, and sensitivity to human review.

Before accepting a benchmark score, test whether the system follows higher-priority instructions, whether search evidence can be replayed, whether automated graders agree with people, and whether human review changes the apparent result.

Takeaway: Audit the current evaluation suite for these failure modes, then pilot only the newly released artifacts relevant to the intended deployment. Treat their publication as a prompt for testing, not proof that their methods are valid.

Another reading: A competing reading is that these are merely newly published proposals whose summaries do not provide independent validation. They may reveal useful evaluation gaps without justifying changes to a production plan beyond a targeted review.

  • S003 records with event kind released: 455
  • S020 records tagged benchmark: 471 count (multi-label)
  • S021 records tagged evaluation: 409 count (multi-label)
  • S023 records tagged agentic: 104 count (multi-label)

What does today's evidence fail to show, and what would change the reading?

high confidence

Today’s captured feed does not establish a field-wide direction, a reliable change from prior days, or independently corroborated metric movement.

There is no certified like-for-like comparison window. Reported download changes are cumulative over each artifact’s full tracked span and come from one source, so they are neither daily changes nor corroborated measures of demand.

Takeaway: The reading would change with a certified comparison window using stable coverage and taxonomy, repeated measurements of the same metric from multiple sources, and independent validation of release claims. Until then, interpret differences as possible collection effects within this keyword-filtered feed.

Another reading: A competing reading is that the captured release volume alone warrants immediate attention. However, release volume in a filtered feed can indicate discovery activity without showing adoption, quality, comparative performance, or a broader field shift.

  • S003 records with event kind released: 455
  • S027 tracked artifacts today seen by more than one data source: 29
  • S028 downloads change for lmarena-ai/leaderboard-dataset: 307,398.0 downloads
  • S029 downloads change for open-llm-leaderboard/requests: -50,181.0 downloads
  • S030 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 39,917.0 downloads
  • S031 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 30,885.0 downloads
  • S032 downloads change for lukakuma/web-trajectories-preference-raw: 26,200.0 downloads
  • S033 downloads change for alexshpunt/explicit-edit-benchmark: 13,667.0 downloads
  • S034 downloads change for RyeAI/danish-asr-leaderboard: 5,194.0 downloads
  • S035 downloads change for lvkaokao/benchmark_data: 5,034.0 downloads
  • S036 daily-average change in benchmark observations: -30.43 observations per day
  • S037 daily-average change in dataset observations: -26.29 observations per day
  • S038 daily-average change in evaluation observations: -6.43 observations per day
  • S039 daily-average change in agentic observations: 2.86 observations per day
  • S040 daily-average change in data_quality observations: -1.14 observations per day

Evidence sources

Where this came from, and what it does not cover

Built from the validated daily snapshot and its 747 evidence records.

Briefing model: gpt-5.6-sol.

It read 178 of 747 records.

This is a keyword-filtered feed, not a representative view of AI research. Only 178 selected evidence records were supplied from today’s 747-record corpus, so the briefing does not reflect every captured artifact. Brave and OpenReview were unavailable, and the comparable history was insufficient to infer category-share movement. Attention and tracked download changes were not used as corroboration because relevant measurements were incomplete or came from single connectors.