Benchmark
NL2Repo Benchmark
What is NL2Repo?
Evaluates generating a repository from a natural language description. Temperature=1.0, top_p=1.0, max_new_tokens=64k under 1M context.
- Released
- 2025-06-15
- Evaluates
- coding agent
- Openness
- unknown
- Importer
- Model reports
- Reported scores
- 3
Source provenance
- Original evidence https://github.com/nl2repo/nl2repo
NL2Repo code
NL2Repo results and reported scores
Model reports
Scores are partitioned by the source that reported them and are never merged into a single cross-source ranking, because the sources measure different things and say so.
Which sources cite NL2Repo?
- DeepSeek-V4.1-Flash DeepSeek, 2026-09-10
- Hy4 preview Tencent, 2026-08-28
- GLM-5.3-Flash Z.ai, 2026-08-27