2025/09/05 by Jessica Lundin, Lundin, Jessica M., Ada Zhang +5
Decision Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #demographic modeling and climate adaptation
paper · pdf · doi:10.48550/arxiv.2509.05425
openalex publication_date 2025/09/05 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/31
We show that translation quality can be predicted with surprising accuracy without ever running the translation system itself. Using only a handful of features, token fertility ratios, token counts, and basic linguistic metadata (language family, script, and region), we can forecast ChrF scores for GPT-4o translations across 203 languages in the FLORES-200 benchmark. Gradient boosting models achieve favorable performance (R2=0.66 for XX→English and R2=0.72 for English→XX). Feature importance analyses reveal that typological factors dominate predictions into English, while fertility plays a larger role for translations into diverse target languages. These findings suggest that translation quality is shaped by both token-level fertility and broader linguistic typology, offering new insights for multilingual evaluation and quality estimation.