Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models

2025/04/04 by Nvidia Nvidia, NVIDIA, : +289 · 39 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation #Computation and Language (cs.CL) #Constant (computer programming) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Inference #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Pruning #Quantization (signal processing) #Recipe #Scaling #Topic Modeling #Transformer

paper · pdf · doi:10.48550/arxiv.2504.03624

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/04/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

As inference-time scaling becomes critical for enhanced reasoning capabilities, it is increasingly becoming important to build models that are efficient to infer. We introduce Nemotron-H, a family of 8B and 56B/47B hybrid Mamba-Transformer models designed to reduce inference cost for a given accuracy level. To achieve this goal, we replace the majority of self-attention layers in the common Transformer model architecture with Mamba layers that perform constant computation and require constant memory per generated token. We show that Nemotron-H models offer either better or on-par accuracy compared to other similarly-sized state-of-the-art open-sourced Transformer models (e.g., Qwen-2.5-7B/72B and Llama-3.1-8B/70B), while being up to 3× faster at inference. To further increase inference speed and reduce the memory required at inference time, we created Nemotron-H-47B-Base from the 56B model using a new compression via pruning and distillation technique called MiniPuzzle. Nemotron-H-47B-Base achieves similar accuracy to the 56B model, but is 20% faster to infer. In addition, we introduce an FP8-based training recipe and show that it can achieve on par results with BF16-based training. This recipe is used to train the 56B model. We are releasing Nemotron-H base model checkpoints with support in Hugging Face and NeMo.

Cited by

• DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
• Are the High-weight Neurons the Important Ones in Image Classification Neural Networks?
• NVIDIA Nemotron 3: Efficient and Open Intelligence
• Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
• InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
• ELANA: A Simple Energy and Latency Analyzer for LLMs
• UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs
• Distributed Dynamic Associative Memory via Online Convex Optimization
• Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models
• Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs
• Apriel-H1: Towards Efficient Enterprise Reasoning Models
• Detecting Data Contamination in LLMs via In-Context Learning
• HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
• EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
• To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Models
• Forget Attention: Importance-Aware Attention Is All You Need
• End-to-End Test-Time Training for Long Context
• Pretraining Large Language Models with NVFP4
• Short window attention enables long-term memorization
• HyMaTE: A Hybrid Mamba and Transformer Model for EHR Representation Learning
• AdaPtis: Reducing Pipeline Bubbles with Adaptive Pipeline Parallelism on Heterogeneous Models
• Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data
• SpikingBrain: Spiking Brain-inspired Large Models
• NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model
• EvoGraph: Hybrid Directed Graph Evolution toward Software 3.0
• SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment
• Characterizing State Space Model (SSM) and SSM-Transformer Hybrid Language Model Performance with Long Context Length
• EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes
• Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
• BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers
• Sampling from Your Language Model One Byte at a Time
• TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embedding
• Cartridges: Lightweight and general-purpose long context representations via self-study
• Training a Scientific Reasoning Model for Chemistry
• Comba: Improving Bilinear RNNs with Closed-loop Control
• Scaling Reasoning without Attention
• AdaSTaR: Adaptive Data Sampling for Training Self-Taught Reasoners
• The Impossibility Triangle of Long-Context Modeling
• Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning

Related

• NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model
• Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
• Zamba: A Compact 7B SSM Hybrid Model
• The Zamba2 Suite: Technical Report
• The Mamba in the Llama: Distilling and Accelerating Hybrid Models