mHC: Manifold-Constrained Hyper-Connections

2025/12/31 by Zhenda Xie, Yixuan Wei, Huanqi Cao +17 · 18 voices · 6 citations
#cs.CL #cs.AI #cs.LG

paper · pdf

Abstract

Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models.

Citations

• DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Language Models are Few-Shot Learners
• Attention Is All You Need
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Densely Connected Convolutional Networks
• Deep Residual Learning for Image Recognition

Cited by

• xHC: Expanded Hyper-Connections
• Attention Residuals
• Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability
• Deep Delta Learning
• Mixture-of-Depths Attention
• Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Decoder-Only Transformers
• From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference
• Acoustic Interference Suppression in Ultrasound images for Real-Time HIFU Monitoring Using an Image-Based Latent Diffusion Model

Discussions

• So we already have the first major paper of 2026, DeepSeek mHC: Manifold-Constrained Hyper-Connections. arxiv.org/pdf/2512.24880 Short thread: this is actually an engineering paper, taking as a starti [bsky, 65 points, 1 comments]
• MHC: Manifold-Constrained Hyper-Connections [hn, 32 points, 3 comments]
• Sinkhorn is back in business! arxiv.org/abs/2512.24880 [bsky, 13 points, 3 comments]
• MHC: Manifold-Constrained Hyper-Connections [hn, 12 points, 1 comments]
• The Manifold Dial: Visualizing Why DeepSeek's mHC Stabilizes Deep Networks [lemmy, 3 points, 0 comments]
• 👀 [bsky, 2 points, 0 comments]
• arxiv.org/abs/2512.24880 [bsky, 2 points, 1 comments]
• This is a 😎 systems optimization by Deepseek! Manifold-Constrained Hyper-Connections arxiv.org/abs/2512.24880 [bsky, 1 points, 0 comments]
• MHC: Manifold-Constrained Hyper-Connections [hn, 1 points, 0 comments]
• arxiv.org/abs/2512.24880 THE WHALE IS BACK BABYYYY [bsky, 1 points, 0 comments]
• arxiv.org/pdf/2512.24880 [bsky, 1 points, 0 comments]
• MHC: Manifold-Constrained Hyper-Connections https://arxiv.org/abs/2512.24880 [bsky, 0 points, 0 comments]
• We bolted a manifold onto the residual stream to stop the net from forgetting itself—then watched the gradient pray to its own reflection. Identity restored, sanity optional. [bsky, 0 points, 0 comments]
• DeepSeek publie un papier en entraînant plus facilement des gros modèles d'IA avec juste un discret surcoût de 7% arxiv.org/pdf/2512.24880 [bsky, 0 points, 0 comments]
• Paper: https://arxiv.org/abs/2512.24880 [bsky, 0 points, 0 comments]
• MHC: Manifold-Constrained Hyper-Connections https://arxiv.org/abs/2512.24880 (https://news.ycombinator.com/item?id=46452172) [bsky, 0 points, 0 comments]
• Manifold-Constrained Hyper-Connections [Xie+, 2026] HC expands Transformer's residual stream width to improve expressivity but breaks the identity mapping. mHC constrains residual mappings to be doubl [bsky, 0 points, 0 comments]
• MHC: Manifold-Constrained Hyper-Connections https:// arxiv.org/abs/2512.24880 # arxiv [mastodon, 0 points, 0 comments]

Related

• mHC-lite: You Don't Need 20 Sinkhorn-Knopp Iterations
• xHC: Expanded Hyper-Connections
• Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning
• Hyper-Connections
• Frac-Connections: Fractional Extension of Hyper-Connections