Outlier-Efficient Hopfield Layers for Large Transformer-Based Models

2024/04/04 by Jerry Yao-Chieh Hu, Pei-Hsuan Chang, Hu, Jerry Yao-Chieh +11 · 2 citations
Materials Science · Physics and Astronomy · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Magnetic Properties and Applications #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.2404.03828

openalex publication_date 2024/04/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce an Outlier-Efficient Modern Hopfield Model (termed OutEffHop) and use it to address the outlier inefficiency problem of training gigantic transformer-based models. Our main contribution is a novel associative memory model facilitating outlier-efficient associative memory retrievals. Interestingly, this memory model manifests a model-based interpretation of an outlier-efficient attention mechanism (\rm Softmax₁): it is an approximation of the memory retrieval process of OutEffHop. Methodologically, this allows us to introduce novel outlier-efficient Hopfield layers as powerful alternatives to traditional attention mechanisms, with superior post-quantization performance. Theoretically, the Outlier-Efficient Modern Hopfield Model retains and improves the desirable properties of standard modern Hopfield models, including fixed point convergence and exponential storage capacity. Empirically, we demonstrate the efficacy of the proposed model across large-scale transformer-based and Hopfield-based models (including BERT, OPT, ViT, and STanHop-Net), benchmarking against state-of-the-art methods like \mathttClipped_Softmax and \mathttGated_Attention. Notably, OutEffHop achieves an average reduction of 22+% in average kurtosis and 26+% in the maximum infinity norm of model outputs across four models. Code is available at \hrefhttps://github.com/MAGICS-LAB/OutEffHopGitHub; models are on \hrefhttps://huggingface.co/collections/magicslabnu/outeffhop-6610fcede8d2cda23009a98fHugging Face Hub; future updates are on \hrefhttps://arxiv.org/abs/2404.03828arXiv.

Cited by

• On the Role of Hidden States of Modern Hopfield Network in Transformer
• Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformers

Related

• Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory
• On the Role of Hidden States of Modern Hopfield Network in Transformer
• Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codes
• Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models
• Nonparametric Modern Hopfield Models