Sample-Efficient Language Modeling with Linear Attention and Lightweight Enhancements

2025/11/04 by Haller, Patrick, Golde, Jonas, Akbik, Alan
Computer Science · #Topic Modeling #Text Readability and Simplification #Natural Language Processing Techniques

paper · doi:10.48550/arxiv.2511.05560

Abstract

We study architectural and optimization techniques for sample-efficient language modeling under the constraints of the BabyLM 2025 shared task. Our model, BLaLM, replaces self-attention with a linear-time mLSTM token mixer and explores lightweight enhancements, including short convolutions, sliding window attention with dynamic modulation, and Hedgehog feature maps. To support training in low-resource settings, we curate a high-quality corpus emphasizing readability and pedagogical structure. Experiments across both STRICT and STRICT-SMALL tracks show that (1) linear attention combined with sliding window attention consistently improves zero-shot performance, and (2) the Muon optimizer stabilizes convergence and reduces perplexity over AdamW. These results highlight effective strategies for efficient language modeling without relying on scale.

Citations

• Lizard: An Efficient Linearization Framework for Large Language Models
• FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
• Practical Efficiency of Muon for Pretraining
• Empirical Evaluation of Knowledge Distillation from Transformers to Subquadratic Language Models
• Muon is Scalable for LLM Training
• Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models
• Sovereign Byte Firewall: Comparing Transformer and Mamba-2 Backbones for Tokenization-Free Zero-Day Network Anomaly Detection
• The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
• Hyena Hierarchy: Towards Larger Convolutional Language Models
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Longformer: The Long-Document Transformer
• Root Mean Square Layer Normalization
• A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics
• Shampoo: Preconditioned Stochastic Tensor Optimization
• Attention Is All You Need
• Adam: A Method for Stochastic Optimization
• Long Short-Term Memory

Related

• ShishuLM: Lightweight Language Model with Hybrid Decoder-MLP Architecture and Paired Weight Sharing
• Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
• Attention and Compression is all you need for Controllably Efficient Language Models
• Efficient Attention Mechanisms for Large Language Models: A Survey
• Simple linear attention language models balance the recall-throughput tradeoff