2024/05/29 by Everlyn Asiko Chimoto, Chimoto, Everlyn Asiko, Jay Gala +9 · 4 citations
Mathematics · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Statistics Education and Methodologies
paper · pdf · doi:10.48550/arxiv.2405.19462
openalex publication_date 2024/05/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Neural Machine Translation models are extremely data and compute-hungry. However, not all data points contribute equally to model training and generalization. Data pruning to remove the low-value data points has the benefit of drastically reducing the compute budget without significant drop in model performance. In this paper, we propose a new data pruning technique: Checkpoints Across Time (CAT), that leverages early model training dynamics to identify the most relevant data points for model performance. We benchmark CAT against several data pruning techniques including COMET-QE, LASER and LaBSE. We find that CAT outperforms the benchmarks on Indo-European languages on multiple test sets. When applied to English-German, English-French and English-Swahili translation tasks, CAT achieves comparable performance to using the full dataset, while pruning up to 50% of training data. We inspect the data points that CAT selects and find that it tends to favour longer sentences and sentences with unique or rare words.