Subword Regularization: Improving Neural Network Translation Models with\n Multiple Subword Candidates

2018/04/29 by Taku Kudo, Kudo, Taku · 58 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1804.10959

openalex publication_date 2018/04/29 · openalex created_date 2022/08/20 · openalex updated_date 2026/07/28

Abstract

Subword units are an effective way to alleviate the open vocabulary problems\nin neural machine translation (NMT). While sentences are usually converted into\nunique subword sequences, subword segmentation is potentially ambiguous and\nmultiple segmentations are possible even with the same vocabulary. The question\naddressed in this paper is whether it is possible to harness the segmentation\nambiguity as a noise to improve the robustness of NMT. We present a simple\nregularization method, subword regularization, which trains the model with\nmultiple subword segmentations probabilistically sampled during training. In\naddition, for better subword sampling, we propose a new subword segmentation\nalgorithm based on a unigram language model. We experiment with multiple\ncorpora and report consistent improvements especially on low resource and\nout-of-domain settings.\n

Cited by

• Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features
• Joint Optimization for Greedy Longest-match Tokenization
• The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages
• Tokens are All You Need: Dual-purpose Semantic IDs for Achieving LLM-Level I/O Efficiency in recommendation systems
• TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
• Enhancing Fully Formatted End-to-End Speech Recognition with Knowledge Distillation via Multi-Codebook Vector Quantization
• AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3
• Digitizing Nepal's Written Heritage: A Comprehensive HTR Pipeline for Old Nepali Manuscripts
• Bolmo: Byteifying the Next Generation of Language Models
• An Empirical Study on Chinese Character Decomposition in Multiword Expression-Aware Neural Machine Translation
• PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generation
• Which Pieces Does Unigram Tokenization Really Need?
• Supplementary Resources and Analysis for Automatic Speech Recognition Systems Trained on the Loquacious Dataset
• What Language is This? Ask Your Tokenizer
• Tokenisation over Bounded Alphabets is Hard
• Fast, memory-efficient genomic interval tokenizers for modern machine learning
• Confounding Factors in Relating Model Performance to Morphology
• Contextual morphologically-guided tokenization for Latin encoder models
• The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages
• L2T-Hyena: Enhancing State-Space Models with an Adaptive Learn-to-Teach Framework
• Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
• IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs
• UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8
• How Different Tokenization Algorithms Impact LLMs and Transformer Models for Binary Code Analysis
• MMQ-v2: Align, Denoise, and Amplify: Adaptive Behavior Mining for Semantic IDs Learning in Recommendation
• Language Models are not Equally Robust to Non-Canonical Tokenization across Languages
• Morpheme Induction for Emergent Language
• Explaining and Mitigating Crosslingual Tokenizer Inequities
• Pctx: Tokenizing Personalized Context for Generative Recommendation
• Forging GEMs: Advancing Greek NLP through Quality-Based Corpus Curation
• See the Text: From Tokenization to Visual Reading
• Back to Bytes: Revisiting Tokenization Through UTF-8
• Lossless Vocabulary Reduction for Auto-Regressive Language Models
• Parallel Tokenizers: Rethinking Encoder Models' Vocabulary Design in Cross-Lingual Transfer of Low-Resource Languages
• Evaluating Embedding Frameworks for Scientific Domain
• Better with Less: Small Proprietary Models Surpass Large Language Models in Financial Transaction Understanding
• Understanding Subword Compositionality of Large Language Models
• Speculating LLMs' Chinese Training Data Pollution from Their Tokens
• False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models
• DecipherGuard: Understanding and Deciphering Jailbreak Prompts for a Safer Deployment of Intelligent Software Systems
• Angular Dispersion Accelerates k-Nearest Neighbors Machine Translation
• Deep learning and abstractive summarisation for radiological reports: an empirical study for adapting the PEGASUS models' family with scarce data
• Whisper Has an Internal Word Aligner
• Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Execution
• MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptation
• Progressive Semantic Residual Quantization for Multimodal-Joint Interest Modeling in Music Recommendation
• SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
• Tokens with Meaning: A Hybrid Tokenization Approach for NLP
• NovoMolGen: Rethinking Molecular Language Model Pretraining
• The impact of tokenizer selection in genomic language models
• SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance
• Special-Character Adversarial Attacks on Open-Source Language Model
• Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
• Optimal Corpus Aware Training for Neural Machine Translation
• Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
• The Art of Breaking Words: Rethinking Multilingual Tokenizer Design
• ASR-Synchronized Speaker-Role Diarization
• SEALGuard: Safeguarding the Multilingual Conversations in Southeast Asian Languages for LLM Software Systems

Related

• Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation
• Adversarial Subword Regularization for Robust Neural Machine Translation
• Subword Segmental Machine Translation: Unifying Segmentation and Target Sentence Generation
• Finding Better Subword Segmentation for Neural Machine Translation
• BPE-Dropout: Simple and Effective Subword Regularization