Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

2018/04/29 by Taku Kudo, Kudo, Taku · 1 voice · 89 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.1804.10959

Accepted as a long paper at ACL2018

arxiv created 2018/04/29 · openalex publication_date 2018/04/29 · arxiv published 2018/04/29 · arxiv updated 2018/05/01 · openalex created_date 2022/08/20 · openalex updated_date 2026/07/28

Abstract

Subword units are an effective way to alleviate the open vocabulary problems in neural machine translation (NMT). While sentences are usually converted into unique subword sequences, subword segmentation is potentially ambiguous and multiple segmentations are possible even with the same vocabulary. The question addressed in this paper is whether it is possible to harness the segmentation ambiguity as a noise to improve the robustness of NMT. We present a simple regularization method, subword regularization, which trains the model with multiple subword segmentations probabilistically sampled during training. In addition, for better subword sampling, we propose a new subword segmentation algorithm based on a unigram language model. We experiment with multiple corpora and report consistent improvements especially on low resource and out-of-domain settings.

Cited by

• Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features
• Joint Optimization for Greedy Longest-match Tokenization
• The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages
• Tokens are All You Need: Dual-purpose Semantic IDs for Achieving LLM-Level I/O Efficiency in recommendation systems
• TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
• Enhancing Fully Formatted End-to-End Speech Recognition with Knowledge Distillation via Multi-Codebook Vector Quantization
• AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3
• Digitizing Nepal's Written Heritage: A Comprehensive HTR Pipeline for Old Nepali Manuscripts
• Bolmo: Byteifying the Next Generation of Language Models
• An Empirical Study on Chinese Character Decomposition in Multiword Expression-Aware Neural Machine Translation
• PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generation
• Which Pieces Does Unigram Tokenization Really Need?
• Supplementary Resources and Analysis for Automatic Speech Recognition Systems Trained on the Loquacious Dataset
• What Language is This? Ask Your Tokenizer
• Tokenisation over Bounded Alphabets is Hard
• Fast, memory-efficient genomic interval tokenizers for modern machine learning
• Confounding Factors in Relating Model Performance to Morphology
• Contextual morphologically-guided tokenization for Latin encoder models
• The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages
• L2T-Hyena: Enhancing State-Space Models with an Adaptive Learn-to-Teach Framework
• Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
• IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs
• UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8
• How Different Tokenization Algorithms Impact LLMs and Transformer Models for Binary Code Analysis
• MMQ-v2: Align, Denoise, and Amplify: Adaptive Behavior Mining for Semantic IDs Learning in Recommendation
• Language Models are not Equally Robust to Non-Canonical Tokenization across Languages
• Morpheme Induction for Emergent Language
• Explaining and Mitigating Crosslingual Tokenizer Inequities
• Pctx: Tokenizing Personalized Context for Generative Recommendation
• Forging GEMs: Advancing Greek NLP through Quality-Based Corpus Curation
• See the Text: From Tokenization to Visual Reading
• Back to Bytes: Revisiting Tokenization Through UTF-8
• Lossless Vocabulary Reduction for Auto-Regressive Language Models
• Parallel Tokenizers: Rethinking Encoder Models' Vocabulary Design in Cross-Lingual Transfer of Low-Resource Languages
• Evaluating Embedding Frameworks for Scientific Domain
• Better with Less: Small Proprietary Models Surpass Large Language Models in Financial Transaction Understanding
• Understanding Subword Compositionality of Large Language Models
• Speculating LLMs' Chinese Training Data Pollution from Their Tokens
• False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models
• DecipherGuard: Understanding and Deciphering Jailbreak Prompts for a Safer Deployment of Intelligent Software Systems
• Angular Dispersion Accelerates k-Nearest Neighbors Machine Translation
• Deep learning and abstractive summarisation for radiological reports: an empirical study for adapting the PEGASUS models' family with scarce data
• Whisper Has an Internal Word Aligner
• Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Execution
• MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptation
• Progressive Semantic Residual Quantization for Multimodal-Joint Interest Modeling in Music Recommendation
• SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
• Tokens with Meaning: A Hybrid Tokenization Approach for NLP
• NovoMolGen: Rethinking Molecular Language Model Pretraining
• The impact of tokenizer selection in genomic language models
• SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance
• Special-Character Adversarial Attacks on Open-Source Language Model
• Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
• Optimal Corpus Aware Training for Neural Machine Translation
• Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
• The Art of Breaking Words: Rethinking Multilingual Tokenizer Design
• ASR-Synchronized Speaker-Role Diarization
• SEALGuard: Safeguarding the Multilingual Conversations in Southeast Asian Languages for LLM Software Systems
• Conditional Unigram Tokenization with Parallel Data
• Tokenizer-Agnostic Engram Module
• Is There a Case for Conversation Optimized Tokenizers in Large Language Models?
• ByteSpan: Information-Driven Subword Tokenisation
• Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights
• Less Is More: Reducing Token Counts Without Compromising Performance
• Sampling from Your Language Model One Byte at a Time
• Detecting Hard-Coded Credentials in Software Repositories via LLMs
• HarMoEny: Efficient Multi-GPU Inference of MoE Models
• One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers
• Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-service
• ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition
• TokAlign: Efficient Vocabulary Adaptation via Token Alignment
• Beyond Text Compression: Evaluating Tokenizers Across Scales
• StochasTok: Improving Fine-Grained Subword Understanding in LLMs
• TalTech Systems for the Interspeech 2025 ML-SUPERB 2.0 Challenge
• The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence
• Is Your LLM Overcharging You? Tokenization, Transparency, and Incentives
• Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility
• BanglaByT5: Byte-Level Modelling for Bangla
• ChemPile: A 250GB Diverse and Curated Dataset for Chemical Foundation Models
• Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications
• An Empirical Study on the Effectiveness of Large Language Models for Binary Code Understanding
• Enhancing Vulnerability Reports with Automated and Augmented Description Summarization
• Inside the LLM Word Factory
• PLDR-LLMs Reason At Self-Organized Criticality
• Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps
• Efficient Multilingual Neural Machine Translation via Corpus-Driven Vocabulary Pruning: An English-Arabic Case Study
• HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization
• MorphTok: Morphologically Grounded Tokenization for Indian Languages
• Limitations of Religious Data and the Importance of the Target Domain: Towards Machine Translation for Guinea-Bissau Creole

Discussions

• Subword Regularization: Improving Neural Network Translation Models [hn, 1 points, 0 comments]

Related

• Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation
• Adversarial Subword Regularization for Robust Neural Machine Translation
• Subword Segmental Machine Translation: Unifying Segmentation and Target Sentence Generation
• Finding Better Subword Segmentation for Neural Machine Translation
• BPE-Dropout: Simple and Effective Subword Regularization