Query-Key Normalization for Transformers

2020/10/08 by Alex Henry, Prudhvi Raj Dachapally, Henry, Alex +5 · 38 citations
Computer Science · Biochemistry, Genetics and Molecular Biology · #Topic Modeling #Natural Language Processing Techniques #Biomedical Text Mining and Ontologies

paper · pdf · doi:10.48550/arxiv.2010.04245

Abstract

Low-resource language translation is a challenging but socially valuable NLP task. Building on recent work adapting the Transformer's normalization to this setting, we propose QKNorm, a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity. Specifically, we apply ℓ₂ normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension. We show improvements averaging 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT'15.

Cited by

• Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
• Raven: High-Recall Sequence Modeling with Sparse Memory Routing
• Next-Embedding Prediction Makes Strong Vision Learners
• DVGT: Driving Visual Geometry Transformer
• VFMF: World Modeling by Forecasting Vision Foundation Model Features
• Long-LRM++: Preserving Fine Details in Feed-Forward Wide-Coverage Reconstruction
• PCMind-2.1-Kaiyuan-2B Technical Report
• Multi-view Pyramid Transformer: Look Coarser to See Broader
• Controlling changes to attention logits
• Selective Rotary Position Embedding
• Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
• Learning to Focus: Focal Attention for Selective and Scalable Transformers
• Arcee Trinity Large Technical Report
• Generative Modeling via Drifting
• DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation
• Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Decoder-Only Transformers
• SeeDNorm: Self-Rescaled Dynamic Normalization
• LongCat-Video Technical Report
• Streaming Generation for Music Accompaniment
• Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos
• VFM-VAE: Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models
• Latent Diffusion Model without Variational Autoencoder
• Generative World Modelling for Humanoids: 1X World Model Challenge Technical Report
• Exact Causal Attention with 10% Fewer Operations
• UniVid: The Open-Source Unified Video Model
• Beyond Outliers: A Study of Optimizers Under Quantization
• ZUNA1.1: A more flexible EEG foundation model for Denoising and Super-resolution
• ELF: Embedded Language Flows
• Tenma: Robust Cross-Embodiment Robot Manipulation with Diffusion Transformer
• Open-sci-ref-0.01: open and reproducible reference baselines for language model and dataset comparison
• FLOWER: Democratizing Generalist Robot Policies with Efficient Vision-Language-Action Flow Policies
• WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling
• Efficient Patent Searching Using Graph Transformers
• Whisfusion: Parallel ASR Decoding with Masked Diffusion
• Matrix-Driven Identification and Reconstruction of LLM Weight Homology
• H3R: Hybrid Multi-view Correspondence for Generalizable 3D Reconstruction
• Motif 2.6B Technical Report
• iLRM: An Iterative Large 3D Reconstruction Model

Related

• On Optimal Transformer Depth for Low-Resource Language Translation
• Less is More! A slim architecture for optimal language translation
• Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCAN
• Enhanced Transformer Architecture for Natural Language Processing
• Attention Link: An Efficient Attention-Based Low Resource Machine Translation Architecture