Benchmark Radar
Before you build an eval: has anyone measured this already?
- The answers are scattered. arXiv, GitHub, dataset pages, model cards, technical reports, leaderboards.
- A name is only the start. What does it measure? Is the code still up? Who reported scores, and under which prompt, tools, and attempts?
- And the pile keeps growing: 2,940 new benchmark releases in 90 days.
Credit assignment: 225 leads, 2 papers
- Ask.Jiayu Wang planned an eval and asked: does it duplicate August 2026 work on credit assignment in agentic training, with small Qwen baselines?
- Search.A coding agent installed the Radar Skill and ran several topic variants.
- Judge.Opening each paper left SRPO and HindSearch: August, credit assignment, small Qwen backbone, released code.
What Benchmark Radar is
- Every record keeps its original source and its score history.
- Open: code on GitHub (pipeline MIT, web UI AGPL-3.0), dataset on Hugging Face (MIT; third-party data keeps its own terms).
Who is viewing or using it
- Researchers at OpenAI, Anthropic, Google, Meta, Microsoft, NVIDIA, Amazon, IBM, ByteDance, and Unitree viewed/using it.
- 300+ users across labs, companies, and universities.
How it works
- DiscoverNew benchmarks are found as they appear in papers, repos, datasets, and leaderboards.
- ExtractEach record keeps its original source, the reported score, who reported it, and the date.
- TrackScore history is followed over time, so new results and new versions stay visible.

Attention
Papers
- This is an association, not a cause. A paper and stars can both come from a well-resourced team.
- For your team: a paper-less benchmark is not wrong, but its setup is harder to check.
Abandoned repos
- A finished benchmark can be pushed once. One-day repos measure activity, not quality.
- A missing repo is a dead citation. Radar now lists gone and archived repos without ranking them.
Saturation

Audits
SWE-bench Pro
public tasks underdetermined (15.0% floor); 3 gold patches fail the benchmark's own verifier.
Published June 21, 2026: 17 days before OpenAI's audit (July 8) estimated ~30% of tasks broken.
Terminal-Bench 2.1
gold-passing tasks still pass after a careless in-workspace deletion.
DeepSWE
reference solutions fail their own verifiers (v1.0). The v1.1 re-grade fixed 3 of them.
- Radar lists 9 audits of 8 benchmarks, all by June Kim: also ProgramBench, MirrorCode, FrontierCode, SlopCodeBench, and Frontier-Bench.
A five-step check
Discover wide
Start from the full landscape, not from what is trending.
Check provenance
Find the paper, the repo, and who reported each score. Confirm the repo still exists.
Check saturation
Ask whether it still separates your candidate models.
Check audits
Look for an independent recheck before you rely on it.
Schedule retirement
Decide in advance when the eval gets dropped, and write the date down.
What is next
- Wider coverage. More sources, more benchmarks, and faster pickup of new releases.
- A deeper audit layer. More audit records linked to the scores they examine.
- Interop with community efforts. Radar's records should flow into shared efforts such as Every Eval Ever, not sit in a silo.
- Better history views. Score movement over time, readable at a glance.
Let your agent draft Related Work
- Install once.One command adds the Benchmark Radar Skill to your coding agent.
- Name your themes.Ask your agent for a Related Work section, one paragraph per theme.
- Get LaTeX and BibTeX.A cited Related Work draft and the missing BibTeX entries, ready to edit.
$ npx skills add ktwu01/benchmark-radar --skill benchmark-radar
Search for it. If it is missing, add it.
- Search.52,120 benchmark records from 41 public sources, one search box.
- Not there?Open the “Submit a benchmark” form on GitHub. English or Chinese is fine.
- Help the next team.Every benchmark you add is one more that others can find.

Get new benchmarks by email
- Open the homepage.The signup box sits next to the search results.
- Pick daily or weekly.Daily brings the latest each day; weekly arrives every Monday.
- Leave anytime.Every email has an unsubscribe link.
- No email needed.The same Daily briefing is on the homepage.


Get involved
- Use the dataset. The full record set is open: huggingface.co/datasets/ktwu01/benchmark-radar
- Contribute a benchmark. If Radar missed one you care about, adding it is a small issue.
- Report an error. A wrong score or a missing source is worth a flag.
- Publish an audit. Radar links it next to the scores it examines.
- Start at benchmark-radar.org. Code and issues: github.com/ktwu01/benchmark-radar
Discord: Benchmark Radar
WeChat: Koutian WuAcknowledgments
June Kim
Independent audits of SWE-bench Pro, Terminal-Bench 2.1, DeepSWE, and five more benchmarks: the evidence behind Finding 5.
Prahitha Movva and Cohere Labs
For hosting this talk in the Open Science Community.
Questions
Discord
WeChat