Benchmark Radar™
Star —

llm-evals-judge-adversarial-benchmark and 6 more in AI benchmark news · Nov 7, 2026

744evidence observations
11sources represented
7public-attention signals

Featured on this day

  1. Tushar29k/llm-evals-judge-adversarial-benchmark
  2. Uncovering the Security Debt of Machine Unlearning in Agentic Deployment
  3. Measuring how AI agents handle everyday wet-lab science
  4. Time and Money in LLM-Generated Code: Benchmark Protocol v1 (Pre-registration)
  5. pradhyut/evidencelock-benchmark
  6. Cross-Dataset Intrusion Detection under Benchmark Shift: Directed Transfer, Probability Calibration, and Alert Burden
  7. A live agent benchmark of independent action authorization for healthcare AI agents

Daily briefing

  1. The newly released LLM Evals Judge Adversarial Benchmark tests the graders themselves. It targets a mock large language model judge and deterministic keyword or regular-expression scorers with negation, keyword echoes, fabricated claims, fake citations, and candidate answers containing instructions or prewritten verdicts for the judge. Why it matters: If you use automated graders to compare models or gate releases, this suite adds a direct check for whether outputs can obtain favorable scores without satisfying the task. That can change which scoring methods you trust and whether high-scoring cases require independent verification. Evidence: E003. High confidence.
  2. The new study “Uncovering the Security Debt of Machine Unlearning in Agentic Deployment” evaluates models after machine unlearning—attempted removal of specified knowledge—inside tool-using agents. Across five methods and two forgotten domains, weight-level certificates did not determine resistance to indirect prompt injection, where malicious instructions arrive through tool content; one method increased executed data-stealing attacks from 34.9% to 42.8%. Why it matters: If you certify an unlearned model for agent deployment, model-level forgetting tests alone do not cover behavior after tools and untrusted content enter the system. The deployment decision therefore needs an agent-level security evaluation using the intended tools and attack paths. Evidence: E022. High confidence.
  3. OpenAI newly released BioBasicsBench, a 100-task benchmark for whether agents can interpret raw experimental data and handle routine wet-lab science tasks. Why it matters: If you are selecting an agent for laboratory support, this introduces a task-level alternative to judging scientific ability from general question answering. It lets the selection process focus on work involving experimental records, although the captured description does not provide the task distribution or scoring details. Evidence: E091. Medium confidence.
  4. The newly deposited “Time and Money in LLM-Generated Code” protocol freezes 20 coding tasks, prompt variants, reference implementations, test sets, deliberately altered programs, model providers, sampling settings, analysis plans, sandbox code, and file hashes before the main experiments; the authors state that no benchmark task had yet been sent to a model. Why it matters: If you need coding comparisons that support audit or replication, this artifact lets you verify that tasks and analysis choices preceded the results. That reduces ambiguity from changing tests, prompts, providers, or analysis after model outputs are known. Evidence: E073. High confidence.
  5. The newly released EVIDENCELOCK benchmark evaluates decisions against evidence that changes through corrections and retractions. Its 297 synthetic cases record source authority, dates, scope, and expected labels, while training, development, and frozen-test partitions are separated by template lineage rather than only by individual example. Why it matters: If you evaluate research assistants or evidence-monitoring systems, this benchmark tests whether they revise conclusions when the underlying record changes, not merely whether they retrieve a supporting passage. Lineage-separated splits also reduce the chance that nearly identical templates appear in training and testing. Evidence: E013. High confidence.
  6. The new cross-dataset intrusion-detection study evaluates transfer among three network-traffic datasets rather than relying on within-dataset tests. Mean area under the receiver operating characteristic curve—ranking attacks above benign traffic—fell from 0.9773 within a domain to 0.5445 across domains, and source-trained probability calibration worsened target-domain probability error. Why it matters: If you are choosing an intrusion detector for a new network, a held-out split from the source benchmark can materially overstate transportability. This design supports a deployment decision based on cross-environment testing, realistic attack prevalence, calibration, and resulting alert burden. Evidence: E021. High confidence.
  7. A newly released healthcare-agent benchmark holds the task, model, prompt, and tools fixed while changing only the action-authorization layer across 576 synthetic care-coordination episodes. Hidden truth differs from the information visible to the authorization gateway, allowing the benchmark to count both unsafe approvals and inappropriate blocks. Why it matters: If you are comparing governance layers for healthcare agents, this controlled design separates the effect of authorization from changes to the underlying agent. Measuring both permitted harm and blocked legitimate work prevents a safety gate from appearing successful merely because it stops more actions overall. Evidence: E064. Medium confidence.

What arrived

What benchmarks, datasets, or evaluation methods did the radar first see today?

medium confidenceNot enough evidence

In this captured, keyword-filtered feed, the first-observed cohort spans overlapping benchmark, dataset, and evaluation tags. Notable release records include an adversarial test of automated graders, EVIDENCELOCK for reasoning over changing evidence, TerraMesh-Masks for Earth-observation segmentation, LeanContext for formal-proof generation, and BioBasicsBench for wet-lab agents.

The arrivals cover several practical testing problems: whether graders can be manipulated, whether systems handle corrected or conflicting evidence, whether models can segment satellite imagery, whether generated mathematical proofs pass a formal checker, and whether agents can interpret laboratory data.

Takeaway: These are representative first sightings, not an exhaustive inventory or proof that the artifacts are new to the field. The supplied evidence packet does not include every artifact first observed today.

Another reading: First observed by the radar can mean delayed discovery rather than a genuinely new release. The feed also includes a repository that explicitly disclaims being a complete benchmark and another that mirrors existing benchmark files, illustrating that keyword matches vary in novelty and substance.

  • S017 records tagged benchmark: 487 count (multi-label)
  • S018 records tagged dataset: 351 count (multi-label)
  • S019 records tagged evaluation: 327 count (multi-label)
  • S022 artifacts first observed by the radar today: 273

Which of today's arrivals document how they score an answer?

medium confidenceNot enough evidence

The clearest cases in this captured feed are the adversarial grader benchmark, which names model-judge and deterministic matching scorers; the WriteTested benchmark, which uses a common editorial rubric; EVIDENCELOCK, which supplies fixed outcome labels; and LeanContext, which checks generated proofs in the original formal verification environment.

These artifacts explain at least the basic decision rule: inspect an answer with an automated judge or text matcher, rate it against the same checklist, compare it with a predefined label, or run a formal checker to see whether a proof is valid.

Takeaway: They provide useful starting points for auditable scoring, but the packet does not expose every threshold, weight, prompt, or aggregation rule. Full artifact documentation and the complete first-observed cohort are needed for an exhaustive answer.

Another reading: A competing reading is that only the adversarial grader benchmark clearly documents answer scoring in the supplied summary. The common rubric points to external methodology, EVIDENCELOCK states target classes without showing the full mapping procedure, and formal proof verification is acceptance checking rather than a graded score.

  • S022 artifacts first observed by the radar today: 273

What is still moving

Which artifacts the radar already tracked moved measurably, and over what span?

medium confidenceNot enough evidence

The registry identifies measurable download movement for the artifacts named by the cited statistics. Each movement is cumulative across that artifact’s stated tracked span, not a one-day change.

The registered records show downloads changing between each artifact’s first and latest tracked observations. The supplied evidence packet has no E citations supporting those artifact-level measurements, so the artifact names and spans cannot be independently validated here.

Takeaway: Use the cited registry entries to render the affected artifacts and their full measurement spans, but treat the result as registry-only within this keyword-filtered feed.

Another reading: The registry is sufficient to rank its registered download changes mechanically. However, without artifact-level E citations, that mechanical ranking does not satisfy the evidence requirement for a fully grounded artifact claim.

  • S025 downloads change for lmarena-ai/leaderboard-dataset: 255,779.0 downloads
  • S026 downloads change for huggingface-projects/drlc-leaderboard-data: 41,626.0 downloads
  • S027 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 37,661.0 downloads
  • S028 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 29,010.0 downloads
  • S029 downloads change for vedangfake/chess-slm-benchmark: 28,763.0 downloads
  • S030 downloads change for alexshpunt/explicit-edit-benchmark: 13,556.0 downloads
  • S031 downloads change for FreshSWE-Py: post-cutoff SWE tasks for Gemma 4: 13,049.0 downloads
  • S032 downloads change for prvsiyan's Extended Name That LLM Dataset: 7,953.0 downloads

Which of that movement is corroborated by more than one data source?

high confidenceNot enough evidence

None of the registered artifact-level download movements is corroborated by more than one data source; each cited movement comes from a single platform.

Seeing an artifact in multiple sources is not the same as having multiple sources measure the same download change. The registry explicitly marks every registered movement here as uncorroborated.

Takeaway: Treat all registered movement as single-source measurement within this captured feed, not independently confirmed movement.

Another reading: The supplied tracked-artifact data contains corroboration flags beyond the registered movement statistics, but those movements lack registry statistics and E citations, so they cannot be reported under the grounding rules.

  • S024 tracked artifacts today seen by more than one data source: 19
  • S025 downloads change for lmarena-ai/leaderboard-dataset: 255,779.0 downloads
  • S026 downloads change for huggingface-projects/drlc-leaderboard-data: 41,626.0 downloads
  • S027 downloads change for khalidsyfullah/privacy-utility-evaluation-results: 37,661.0 downloads
  • S028 downloads change for Agnuxo/P2PCLAW-Innovative-Benchmark: 29,010.0 downloads
  • S029 downloads change for vedangfake/chess-slm-benchmark: 28,763.0 downloads
  • S030 downloads change for alexshpunt/explicit-edit-benchmark: 13,556.0 downloads
  • S031 downloads change for FreshSWE-Py: post-cutoff SWE tasks for Gemma 4: 13,049.0 downloads
  • S032 downloads change for prvsiyan's Extended Name That LLM Dataset: 7,953.0 downloads

What it means

What should someone building or evaluating AI systems do differently today?

medium confidence

In this captured, keyword-filtered feed, recent daily averages rose across the overlapping benchmark, dataset, and evaluation labels in comparable windows. New releases highlight adversarial attacks on automated graders, system-level testing for agents, corrected evaluators, and reproducibility materials.

Review evaluation gates rather than adding every new benchmark. Test whether outputs can trick the grading system, and for agents, test tool use, retained state, outside-service calls, and human handoffs. Prefer artifacts that include replication scripts, audit records, and clearly defined test splits.

Takeaway: Add a targeted evaluation review today: identify grader-gaming risks, cover whole-system agent behavior where relevant, and require reproducible evidence before a new benchmark influences release decisions.

Another reading: The highlighted releases are heterogeneous and domain-specific. Teams that do not use automated judges or action-taking agents may reasonably make no immediate process change, especially because the feed does not independently validate the releases.

  • S033 daily-average change in dataset observations: 59.0 observations per day
  • S034 daily-average change in benchmark observations: 53.71 observations per day
  • S035 daily-average change in evaluation observations: 38.57 observations per day

What does today's evidence fail to show, and what would change the reading?

high confidence

This captured feed does not show a representative field-wide shift, independently verified benchmark quality, or broad corroboration. Only a small subset of tracked artifacts was seen by more than one source, and the composition check found no material persistent category-share pattern.

More records do not establish that systems improved or that any benchmark is trustworthy. The feed lacks independent reruns, consistent quality checks, and enough matching reports from separate sources to distinguish meaningful change from publication and collection activity.

Takeaway: The reading would change with independent reproductions, matching measurements from multiple sources, documented contamination and data-quality checks, and sustained patterns under stable connector coverage.

Another reading: Some releases provide code, audit files, replication scripts, corrected evaluation tools, and uncertainty analysis. Those materials make verification possible, but the packet does not show that independent parties completed it.

  • S001 evidence records captured today: 744
  • S024 tracked artifacts today seen by more than one data source: 19
  • S033 daily-average change in dataset observations: 59.0 observations per day
  • S034 daily-average change in benchmark observations: 53.71 observations per day
  • S035 daily-average change in evaluation observations: 38.57 observations per day
  • S036 daily-average change in agentic observations: 4.86 observations per day
  • S037 daily-average change in data_quality observations: 0.57 observations per day

Evidence sources

Where this came from, and what it does not cover

Built from the validated daily snapshot and its 744 evidence records.

Briefing model: gpt-5.6-sol.

It read 146 of 744 records.

This is a keyword-filtered feed, not a representative sample of artificial intelligence work. Only 146 of 744 corpus evidence records were injected into this briefing, so it does not reflect a full reading of the day’s corpus. Brave and OpenReview were unavailable, and several attention collectors had failed requests or stale carry-forwards; no attention metric was used as evidence above.