2025/05/16 by Xianglong Xu, Xu, Xianglong, John W. Bowen +3
Computer Science · #Artificial Intelligence (cs.AI) #Authorship Attribution and Profiling #Computation and Language (cs.CL) #FOS: Computer and information sciences #Sentiment Analysis and Opinion Mining #Text and Document Classification Technologies
paper · pdf · doi:10.48550/arxiv.2505.11746
openalex publication_date 2025/05/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
While transformer-based models achieve strong performance on text classification, we explore whether masking input tokens can further enhance their effectiveness. We propose token masking regularization, a simple yet theoretically motivated method that randomly replaces input tokens with a special [MASK] token at probability p. This introduces stochastic perturbations during training, leading to implicit gradient averaging that encourages the model to capture deeper inter-token dependencies. Experiments on language identification and sentiment analysis -- across diverse models (mBERT, Qwen2.5-0.5B, TinyLlama-1.1B) -- show consistent improvements over standard regularization techniques. We identify task-specific optimal masking rates, with p = 0.1 as a strong general default. We attribute the gains to two key effects: (1) input perturbation reduces overfitting, and (2) gradient-level smoothing acts as implicit ensembling.