Benchmark Radar: A Living Database and Search Engine for AI Benchmarks and Evaluation

Why it exists

I kept running into new benchmarks while doing benchmark research. There was no one place that listed them, said what each one tests, and showed which models had been measured on it. So I built a collector that reads public sources every day and writes down what it finds, with the evidence attached.

What it collects

Two things, kept apart. Every day the collector reads arXiv, GitHub, Hugging Face, OpenReview, Semantic Scholar, Hacker News, and the labs' own release feeds, and writes down the benchmark work that appeared that day. Separately, the scored catalog is built from registered leaderboards and model reports: OpenCompass Hub, LLM Stats, Artificial Analysis, and the evaluation tables labs publish with a model. Each benchmark in that catalog keeps its own page with the paper, code, and dataset links it has, plus every score on record.

The rule the whole project follows

A number is only worth as much as the document it came from. Scores stay partitioned by the source that reported them and are never merged into a single ranking, because those sources measure under different conditions and say so. Where a field is missing, the page leaves it out rather than filling it in.

Who makes it

Benchmark Radar is built by Koutian Wu and open to contributions. The code is MIT-licensed, the data is downloadable in full, and every page is generated from the same public corpus, so anyone can check a claim against its source.

Where to start

• Browse every benchmark in the catalog, one page each
• Compare reported scores across the benchmark frontier
• Watch scores on a benchmark climb toward saturation
• Read the daily brief: what appeared today and what it means
• Query the whole catalog offline from the command line
• Cite the technical report, or read the publications behind it

Benchmark Radar elsewhere

• Source code on GitHub The collector, the catalog contract, the site build, and the issue tracker.
• Technical report How the catalog is collected, scored, and kept current, written up on arXiv.
• Download the full dataset The benchmark catalog, detail records, and daily discovery snapshots in one ZIP.
• Build log: every day of this project, written up Koutian Wu's blog, with a post for each day of work from day 1 onward.
• Benchmark Radar: an evidence-first daily radar for AI benchmarks The post that started it, explaining the idea before any of it was built.
• Why hard benchmarks should not become coding tricks What goes wrong when a benchmark rewards the harness instead of the ability.
• The value of benchmarks and the ability to frame questions Whether building benchmarks counts as research, argued out in full.
• Google Scholar Other work by the author.