QuechuaTok Benchmark
Explore score history ↗A comparison of tokenization strategies (BPE, Unigram LM, WordPiece, PRPE) for Southern Quechua using a 200k-sentence corpus and a finite-state morphological analyzer as reference, with metrics including fertility rate, OOV rate, and morphological boundary accuracy.
- Released
- 2026-06-22
- Institutions
- Peruvian University of Applied Sciences
- Evaluates
- Language & Knowledge, Software & AI Compute, cs.CL
- Openness
- unknown
- Importer
- Claire Radar
- Review status
- unreviewed
QuechuaTok paper
No reported scores are on record for this benchmark yet.