Validating open-source machine translation for quantitative text analysis

2024/02/07 by Hauke Licht, Ronja Sczepanski, Moritz Laurer +1 · 1 voice
Computer Science · #Natural Language Processing Techniques

paper · pdf · doi:10.31219/osf.io/9trjs

Abstract

Machine translation (MT) is an essential tool in many multilingual computational text analysis applications. However, relying on commercial services like Google Translate or DeepL limits reproducibility and can be expensive. This paper assesses the viability of a reproducible, transparent, and affordable alternative: open-source MT models. We ask whether using open-source MT models instead of commercial services substantially changes the measurements obtained from multilingual corpora by extending an influential study by de Vries et al. (2018) and contributing an original study focusing on Transformer-based supervised text classification. Our findings reveal negligible differences in results between the two MT approaches, suggesting that open-source MT models are highly valuable tools for multilingual text analysis.

Citations

• Positions and saliency of immigration in party manifestos: A novel dataset using crowd coding
• The Evolution of the Immigration Debate: Evidence from a New Dataset of Party Positions Over the Last Half-Century
• ChatGPT outperforms crowd workers for text-annotation tasks

Discussions

• Do you want to translate texts for a quantitative analysis, but using Google Translate & Co. is too expensive? Then use an open-source translation model instead – @haukelicht.bsky.social , Moritz Lau [bsky, 24 points, 1 comments]

Related

• Machine Translation for Accessible Multi-Language Text Analysis
• Languages Still Left Behind: Toward a Better Multilingual Machine Translation Benchmark
• MTUncertainty: Assessing the Need for Post-editing of Machine Translation Outputs by Fine-tuning OpenAI LLMs
• Machine Translation: A Literature Review
• On the Influence of Machine Translation on Language Origin Obfuscation