NL2Scratch Benchmark
Explore score history ↗NL2Scratch is an executable benchmark for natural-language-to-Scratch generation, consisting of 311,648 parser-valid NL-program pairs extracted from real Scratch projects. It includes a semantically validated pool of 23,594 examples and an 800-example diagnostic set. Evaluation uses Semantic Alignment Consistency (SAC), an interpretable slot-level metric for measuring semantic agreement.
- Released
- 2026-06-20
- Institutions
- ETH Zurich
- Evaluates
- Code & Software, Code generation, General AI, cs.CL
- Openness
- unknown
- Importer
- Claire Radar
- Review status
- unreviewed
NL2Scratch paper
No reported scores are on record for this benchmark yet.