2025/04/16 by M. Baig, Baig, Mirza Samad Ahmed, Syeda Anshrah Gillani +6
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Adversarial system #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Gaussian #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Range (aeronautics) #Regularization (linguistics) #Robustness (evolution) #Smoothing #Stability (learning theory) #Transformer
paper · pdf · doi:10.48550/arxiv.2504.12088
openalex publication_date 2025/04/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Transformer-based architectures achieve state-of-the-art performance across a wide range of tasks in natural language processing, computer vision, and speech processing. However, their immense capacity often leads to overfitting, especially when training data is limited or noisy. In this research, a unified family of stochastic regularization techniques has been proposed, i.e. AttentionDrop with its three different variants, which operate directly on the self-attention distributions. Hard Attention Masking randomly zeroes out top-k attention logits per query to encourage diverse context utilization, Blurred Attention Smoothing applies a dynamic Gaussian convolution over attention logits to diffuse overly peaked distributions, and Consistency-Regularized AttentionDrop enforces output stability under multiple independent AttentionDrop perturbations via a KL-based consistency loss. Results achieved in the study demonstrate that AttentionDrop consistently improves accuracy, calibration, and adversarial robustness over standard Dropout, DropConnect, and R-Drop baselines