Transformers without Normalization

2025/03/13 by Jiachen Zhu, Zhu, Jiachen, Xinlei Chen +7 · 20 voices · 46 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Artificial neural network #Generative Adversarial Networks and Image Synthesis #Normalization (sociology) #Pattern recognition (psychology) #Preprocessor #Transformer #cs.AI #cs.CL #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2503.10622

openalex publication_date 2025/03/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Normalization layers are ubiquitous in modern neural networks and have long been considered essential. This work demonstrates that Transformers without normalization can achieve the same or better performance using a remarkably simple technique. We introduce Dynamic Tanh (DyT), an element-wise operation DyT(x) = \tanh(αx), as a drop-in replacement for normalization layers in Transformers. DyT is inspired by the observation that layer normalization in Transformers often produces tanh-like, S-shaped input-output mappings. By incorporating DyT, Transformers without normalization can match or exceed the performance of their normalized counterparts, mostly without hyperparameter tuning. We validate the effectiveness of Transformers with DyT across diverse settings, ranging from recognition to generation, supervised to self-supervised learning, and computer vision to language models. These findings challenge the conventional understanding that normalization layers are indispensable in modern neural networks, and offer new insights into their role in deep networks.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Were RNNs All We Needed?
• InternLM2 Technical Report
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Simplifying Transformer Blocks
• Mistral 7B
• HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution
• A ConvNet for the 2020s
• MLP-Mixer: An all-MLP Architecture for Vision
• High-Performance Large-Scale Image Recognition Without Normalization
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Group Normalization
• Attention Is All You Need
• Self-Normalizing Neural Networks
• Deep Networks with Stochastic Depth
• Deep Residual Learning for Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Cited by

• Sharp Stability Threshold and Certification for Designing Stable Residual Architectures
• Edge-Efficient Transformer for End-to-End RF Spectrum Monitoring
• A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
• Beyond blind feature injection: An information foraging theory-guided deep learning framework for product recommendation
• Stronger Normalization-Free Transformers
• Discontinuity-aware physics-informed neural network for phase-field method in three-phase flow with phase change
• SCALAR: Benchmarking SAE Interaction Sparsity in Toy LLMs
• SeeDNorm: Self-Rescaled Dynamic Normalization
• Spatially Aware Linear Transformer (SAL-T) for Particle Jet Tagging
• Chain-of-Influence: Tracing Interdependencies Across Time and Features in Clinical Predictive Modelings
• A Mathematical Explanation of Transformers for Large Language Models and GPTs
• GateMABSA: Aspect-Image Gated Fusion for Multimodal Aspect-based Sentiment Analysis
• Graph Neural Networks with Diversity-aware Neighbor Selection and Dynamic Multi-scale Fusion for Multivariate Time Series Forecasting
• Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformers
• Every Subtlety Counts: Fine-grained Person Independence Micro-Action Recognition via Distributionally Robust Optimization
• EfficienT-HDR: An Efficient Transformer-Based Framework via Multi-Exposure Fusion for HDR Reconstruction
• FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillation
• DyGLNet: Hybrid Global-Local Feature Fusion with Dynamic Upsampling for Medical Image Segmentation
• Entanglement and optimization within autoregressive neural quantum states
• Fractal Flow: Hierarchical and Interpretable Normalizing Flow via Topic Modeling and Recursive Strategy
• On Surjectivity of Neural Networks: Can you elicit any behavior from your model?
• E-CaTCH: Event-Centric Cross-Modal Attention with Temporal Consistency and Class-Imbalance Handling for Misinformation Detection
• Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classification
• DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD
• Discontinuity-aware KAN-based physics-informed neural networks
• Mamba Goes HoME: Hierarchical Soft Mixture-of-Experts for 3D Medical Image Segmentation
• Multi-Level Fusion Graph Neural Network for Molecule Property Prediction
• Accurate Parameter-Efficient Test-Time Adaptation for Time Series Forecasting
• Subspace-Boosted Model Merging
• MGDFIS: Multi-scale Global-detail Feature Integration Strategy for Small Object Detection
• Machine Intelligence on Wireless Edge Networks
• Trajectory First: A Curriculum for Discovering Diverse Policies
• ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code
• Taming LLMs by Scaling Learning Rates with Gradient Grouping
• Weight-Space Linear Recurrent Neural Networks
• Learning in Compact Spaces with Approximately Normalized Transformer
• Expressive yet Efficient Feature Expansion with Adaptive Cross-Hadamard Products
• Towards Fully FP8 GEMM LLM Training at Scale
• Pangu Light: Weight Re-Initialization for Pruning and Accelerating LLMs
• AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping
• Temporal and Spatial Feature Fusion Framework for Dynamic Micro Expression Recognition
• Guarded Query Routing for Large Language Models
• Just One Layer Norm Guarantees Stable Extrapolation
• SymNet: A Multi-Task Network for Joint Radio Map Reconstruction and Transmitter Localization
• Statistical CSI-Based Distributed Precoding Design for OFDM-Cooperative Multi-Satellite Systems
• Sphere Retraction Normalizations

Discussions

• Wow, this seems to be extremely easy to code and extremely useful. Transformers without Normalization Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, Zhuang Liu arxiv.org/abs/2503.10622 [bsky, 50 points, 2 comments]
• Transformers without normalization [hn, 42 points, 6 comments]
• arxiv.org/abs/2503.10622 good paper [bsky, 8 points, 2 comments]
• Transformers without Normalization [lobsters, 5 points, 1 comments]
• Transformers Without Normalization [hn, 4 points, 0 comments]
• arxiv.org/abs/2503.10622 [bsky, 3 points, 0 comments]
• Dynamic Tanh sounds kinda cool. arxiv.org/abs/2503.10622 [bsky, 2 points, 0 comments]
• Transformers Without Normalization [hn, 2 points, 0 comments]
• Transformers Without Normalization [hn, 2 points, 1 comments]
• CDS founding director Yann LeCun (@yann-lecun.bsky.social) & NYU Courant PhD student Jiachen Zhu show that transformers can achieve excellent results without normalization layers. Their "Dynamic Tanh" [bsky, 1 points, 0 comments]
• Transformers Without Normalization https://arxiv.org/abs/2503.10622 [bsky, 0 points, 0 comments]
• Transformers without Normalization arxiv.org/abs/2503.10622 Project page: jiachenzhu.github.io/DyT/ #CVPR2025 🎉 [bsky, 0 points, 0 comments]
• https://arxiv.org/abs/2503.10622 Transformerの正規化層は必須と考えられていましたが、この研究はそれを覆します。 Dynamic Tanh (DyT) というシンプルな手法で、正規化なしで同等以上の性能を達成できることを示しました。 DyTは、認識から生成、教師あり学習から自己教師あり学習まで、幅広い設定で有効であることが確認されています。 [bsky, 0 points, 0 comments]
• Transformers without Normalization arxiv.org/pdf/2503.10622 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2503.10622 tanh is all you need? [bsky, 0 points, 0 comments]
• Normalization-Free Transformers rediscover controlled signal propagation — a core idea behind SNNs. P: arxiv.org/abs/2503.10622 [bsky, 0 points, 1 comments]
• [2503.10622] Transformers without Normalization arxiv.org/abs/2503.10622 [bsky, 0 points, 0 comments]
• Paper: arxiv.org/abs/2503.10622 Project page: jiachenzhu.github.io/DyT/ FAIR Meta @yann-lecun.bsky.social @nyu.edu [bsky, 0 points, 0 comments]
• Interesting: one can train deep nets without #normalization layers by replacing them with a parameterized tanh() arxiv.org/abs/2503.10622 tanh() are my favourite #activation function #deeplearning #ar [bsky, 0 points, 0 comments]
• Transformers without Normalization https://lobste.rs/s/b784dz #pdf #ai [bsky, 0 points, 0 comments]

Related

• Stronger Normalization-Free Transformers
• DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD
• SeeDNorm: Self-Rescaled Dynamic Normalization
• HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization
• On Separate Normalization in Self-supervised Transformers