RecurrentGemma: Moving Past Transformers for Efficient Open Language Models

2024/04/11 by Aleksandar Botev, Botev, Aleksandar, Soham De +127 · 1 voice · 14 citations
Computer Science · Engineering · #Architecture #Artificial intelligence #Computer science #Embedding #Engineering #Griffin #Inference #Language model #Natural Language Processing Techniques #Natural language processing #Programming language #Speech Recognition and Synthesis #Theoretical computer science #Topic Modeling #Transformer

paper · pdf · doi:10.48550/arxiv.2404.07839

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/04/11 · openalex created_date 2024/04/13 · openalex updated_date 2026/07/28

Abstract

We introduce RecurrentGemma, a family of open language models which uses Google's novel Griffin architecture. Griffin combines linear recurrences with local attention to achieve excellent performance on language. It has a fixed-sized state, which reduces memory use and enables efficient inference on long sequences. We provide two sizes of models, containing 2B and 9B parameters, and provide pre-trained and instruction tuned variants for both. Our models achieve comparable performance to similarly-sized Gemma baselines despite being trained on fewer tokens.

Cited by

• Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search
• Some Attention is All You Need for Retrieval
• Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models
• Forget Attention: Importance-Aware Attention Is All You Need
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
• Linear RNNs for autoregressive generation of long music samples
• Mamba Modulation: On the Length Generalization of Mamba
• HistoryBankQA: Multilingual Temporal Question Answering on Historical Events
• DySK-Attn: A Framework for Efficient, Real-Time Knowledge Updating in Large Language Models via Dynamic Sparse Knowledge Attention
• A Systematic Analysis of Hybrid Linear Attention
• LoLA: Low-Rank Linear Attention With Sparse Caching
• Overflow Prevention Enhances Long-Context Recurrent LLMs
• Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation

Discussions

• RecurrentGemma: Moving Past Transformers for Efficient Open Language Models [hn, 47 points, 3 comments]

Related

• Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
• GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling
• Simple Recurrence Improves Masked Language Models
• Retentive Network: A Successor to Transformer for Large Language Models
• Autoregressive + Chain of Thought = Recurrent: Recurrence's Role in Language Models' Computability and a Revisit of Recurrent Transformer