What benchmarks, datasets, or evaluation methods did the radar first see today?
medium confidenceNot enough evidence
Within this captured feed, first-seen releases covered multi-stage medical agents, crisis communication, cross-benchmark agent results, clarification-aware orchestration, automated retrieval-system development, execution safety, and contract-based web behavior. They also introduced proof-verified benchmark rewriting and a judge-scale reversal probe. A separately discovered record proposed a resampling-based test for comparing automated judges with human annotators.
Today’s first-seen material tests systems in more realistic workflows: changing clinical cases, public crises, underspecified requests, tool execution, and interactive software. Other arrivals focus on making evaluation itself more trustworthy by checking rewritten answers with formal proofs, exposing scoring-scale bias, or statistically comparing machine graders with people.
Takeaway: The registry confirms a substantial first-seen cohort, but the available evidence supports only a representative map rather than a complete inventory. These are first observations by this keyword-filtered radar, not proof that the artifacts are new to the field.
Another reading: The packet does not include full descriptions for every first-seen artifact, and some records were discoveries rather than releases. Collection changes and unavailable sources could also shape what appeared today, so the mix should not be interpreted as a field-wide shift.
- S023 artifacts first observed by the radar today: 286
Which of today's arrivals document how they score an answer?
medium confidenceNot enough evidence
The clearest examples are TraceProbe, which scores rule violations from execution traces; RePro, which validates regenerated mathematical answers with formal proofs; a weight-editing benchmark that scores held-out success, side effects, and accuracy loss; and Sampling Config Benchmark, which uses unit tests for coding and repair tasks. A classification repository explicitly lists accuracy, precision, recall, error-balancing scores, confusion matrices, and operating-characteristic curves.
These arrivals do more than say whether an output looks good. They check observable behavior: whether actions break rules, whether a mathematical answer can be formally proved, whether edited models still work on unseen images, whether generated code passes tests, and whether predictions match labeled outcomes.
Takeaway: Several arrivals expose concrete grading mechanisms, especially executable tests, formal verification, trace-based rules, and held-out measurements. However, the summaries do not provide complete rubrics, aggregation rules, thresholds, or tie handling for every arrival, so this is a confirmed subset rather than an exhaustive answer.
Another reading: Some named mechanisms evaluate trajectories, model edits, distributions, or classifiers rather than a single free-text answer. The available summaries may also overstate documentation quality because the full scoring implementations and protocols were not inspected.
- S023 artifacts first observed by the radar today: 286