Data Engineering for Scaling Language Models to 128K Context

2024/02/15 by Yao Fu, Rameswar Panda, Fu, Yao +11 · 39 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Network Packet Processing and Optimization #Parallel Computing and Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2402.10171

openalex publication_date 2024/02/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We study the continual pretraining recipe for scaling language models' context lengths to 128K, with a focus on data engineering. We hypothesize that long context modeling, in particular the ability to utilize information at arbitrary input locations, is a capability that is mostly already acquired through large-scale pretraining, and that this capability can be readily extended to contexts substantially longer than seen during training~(e.g., 4K to 128K) through lightweight continual pretraining on appropriate data mixture. We investigate the quantity and quality of the data for continual pretraining: (1) for quantity, we show that 500 million to 5 billion tokens are enough to enable the model to retrieve information anywhere within the 128K context; (2) for quality, our results equally emphasize domain balance and length upsampling. Concretely, we find that naively upsampling longer data on certain domains like books, a common practice of existing work, gives suboptimal performance, and that a balanced domain mixture is important. We demonstrate that continual pretraining of the full model on 1B-5B tokens of such data is an effective and affordable strategy for scaling the context length of language models to 128K. Our recipe outperforms strong open-source long-context models and closes the gap to frontier models like GPT-4 128K.

Cited by

• AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees
• Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
• LongWeave: A Long-Form Generation Benchmark Bridging Real-World Relevance and Verifiability
• Efficient Long-context Language Model Training by Core Attention Disaggregation
• UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models
• Revisiting Long-context Modeling from Context Denoising Perspective
• Contrastive Retrieval Heads Improve Attention-Based Re-Ranking
• Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
• When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models
• Long Context Automated Essay Scoring with Language Models
• LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation
• ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning
• Technical Report of TeleChat2, TeleChat2.5 and T1
• Towards Compute-Optimal Many-Shot In-Context Learning
• LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models
• PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning
• From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation
• QFFT, Question-Free Fine-Tuning for Adaptive Reasoning
• SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks
• Don't Pay Attention
• Unleashing Hour-Scale Video Training for Long Video-Language Understanding
• SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models
• A Controllable Examination for Long-Context Language Models
• Curse of High Dimensionality Issue in Transformer for Long-context Modeling
• Automated Essay Scoring Incorporating Annotations from Automated Feedback Systems
• Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaboration
• MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models
• Monocle: Hybrid Local-Global In-Context Evaluation for Long-Text Generation with Uncertainty-Based Active Learning
• QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning
• LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
• SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization
• LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis
• CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capability
• Scaling Context, Not Parameters: Training a Compact 7B Language Model for Efficient Long-Context Processing
• Formal Verification of TurboQuant: Machine-Checked Proofs and Gap Closures
• MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scores
• LUCid: Redefining Relevance For Lifelong Personalization
• From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
• Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillation

Related

• From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
• Effective Long-Context Scaling of Foundation Models
• Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation
• Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models
• How to Train Long-Context Language Models (Effectively)