The Super Weight in Large Language Models

2024/11/11 by Mengxia Yu, Wang De, Yu, Mengxia +9 · 19 voices · 10 citations
Computer Science · #Natural Language Processing Techniques #Topic Modeling #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2411.07191

openalex publication_date 2024/11/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent works have shown a surprising result: a small fraction of Large Language Model (LLM) parameter outliers are disproportionately important to the quality of the model. LLMs contain billions of parameters, so these small fractions, such as 0.01%, translate to hundreds of thousands of parameters. In this work, we present an even more surprising finding: Pruning as few as a single parameter can destroy an LLM's ability to generate text -- increasing perplexity by 3 orders of magnitude and reducing zero-shot accuracy to guessing. We propose a data-free method for identifying such parameters, termed super weights, using a single forward pass through the model. We additionally find that these super weights induce correspondingly rare and large activation outliers, termed super activations. When preserved with high precision, super activations can improve simple round-to-nearest quantization to become competitive with state-of-the-art methods. For weight quantization, we similarly find that by preserving the super weight and clipping other weight outliers, round-to-nearest quantization can scale to much larger block sizes than previously considered. To facilitate further research into super weights, we provide an index of super weight coordinates for common, openly available LLMs.

Cited by

• Double-Scoring: Reliable Extraction of Strong Lottery Tickets
• SparseSwaps: Tractable LLM Pruning Mask Refinement at Scale
• CryptoTensors: A Light-Weight Large Language Model File Format for Highly-Secure Model Distribution
• Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance
• Don't Be Greedy, Just Relax! Pruning LLMs via Frank-Wolfe
• The Achilles' Heel of LLMs: How Altering a Handful of Neurons Can Cripple Language Abilities
• Stability of Transformers under Layer Normalization
• Probability Signature: Bridging Data Semantics and Embedding Structure in Language Models
• Exploring and Reshaping the Weight Distribution in LLM
• Unveiling Super Experts in Mixture-of-Experts Large Language Models

Discussions

• *removes a single neuron, destroying your ability to think* nothing personnel kid arxiv.org/abs/2411.07191 [bsky, 166 points, 8 comments]
• In the first example of sandbagging, LLMs trick researchers into believing in a fake off-switch: arxiv.org/abs/2411.07191 [bsky, 43 points, 3 comments]
• #ai, #ml or #llm people here, what do you think about the “super weight” paper? TLDR: deleting one single weight from a 7B model turns it completely incoherent, destroying it’s ability to generate le [bsky, 35 points, 3 comments]
• w h a t "Pruning as few as a single parameter can destroy an LLM's ability to generate text -- increasing perplexity by 3 orders of magnitude and reducing zero-shot accuracy to guessing." [bsky, 34 points, 1 comments]
• Researchers identifies "Super Weights" which are at most a handful of weights (amongst billions!). Models end up generating complete gibberish if these weights are removed, underlining the brittleness [bsky, 17 points, 2 comments]
• The Super Weight in Large Language Models Setting as few as a single weight to zero will make various LLMs go from generating coherent text to outputting gibberish. arxiv.org/abs/2411.07191 [bsky, 11 points, 2 comments]
• Crazy finding: In (some?) neural networks there exist a super-weight whose pruning can destroy model's ability to generate text. One should really wonder if this surveys through reproductions. arxiv.o [bsky, 7 points, 0 comments]
• The Super Weight in Large Language Models [hn, 6 points, 0 comments]
• arxiv.org/abs/2411.07191 was in the trend in X, but I think it's well-known and was explored in the context of adversarial training dl.acm.org/doi/10.5555/... [bsky, 6 points, 0 comments]
• The Super Weight in Large Language Models [hn, 4 points, 0 comments]
• The Super Weight in Large Language Models [hn, 3 points, 0 comments]
• *The Super Weight in Large Language Models* by Yu et al. Identifies single weights in LLMs that destroy inference when deactivated. Tracks their mechanisms through the LLM and proposes quantization-s [bsky, 2 points, 0 comments]
• I actually was waiting for just this. This is the Dvoretzky theorem in action. arxiv.org/abs/2411.07191 [bsky, 2 points, 0 comments]
• Pruning a single weight in an LLM can destroy its output [hn, 1 points, 0 comments]
• reminds me of arxiv.org/abs/2411.07191! [bsky, 1 points, 1 comments]
• Intriguing... arxiv.org/pdf/2411.07191 [bsky, 1 points, 0 comments]
• A study reveals that zeroing a single “super weight” in an LLM can drastically degrade its performance. The authors propose a method for identifying such weights in one forward pass. Article: bit.ly/ [bsky, 0 points, 0 comments]
• We’re covering this one tonight at #CognitionTO Superweighty. I love this paper. arxiv.org/abs/2411.07191 [bsky, 0 points, 0 comments]
• The super weight in LLMs: arxiv.org/abs/2411.07191 Massive Activations in LLMs: arxiv.org/abs/2402.17762 [bsky, 0 points, 0 comments]

Related

• Systematic Outliers in Large Language Models
• House of Cards: Massive Weights in LLMs
• Rethinking the Outlier Distribution in Large Language Models: An In-depth Study
• Data-free Weight Compress and Denoise for Large Language Models
• GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs