Effective Long-Context Scaling of Foundation Models

2023/09/27 by Wenhan Xiong, Jingyu Liu, Xiong, Wenhan +40 · 2 voices · 75 citations
Computer Science · #Artificial intelligence #Computer science #Context (archaeology) #Human–computer interaction #Key (lock) #Language model #Machine learning #Natural Language Processing Techniques #Natural language processing #Process (computing) #Programming language #Scratch #Speech Recognition and Synthesis #Suite #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2309.16039

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/09/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual pretraining from Llama 2 with longer training sequences and on a dataset where long texts are upsampled. We perform extensive evaluation on language modeling, synthetic context probing tasks, and a wide range of research benchmarks. On research benchmarks, our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2. Notably, with a cost-effective instruction tuning procedure that does not require human-annotated long instruction data, the 70B variant can already surpass gpt-3.5-turbo-16k's overall performance on a suite of long-context tasks. Alongside these results, we provide an in-depth analysis on the individual components of our method. We delve into Llama's position encodings and discuss its limitation in modeling long dependencies. We also examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths -- our ablation experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.

Cited by

• Enhancing Vision Foundation Models via Multimodal Continual Pre-Training
• StabilityBench: Benchmarking Instability in LLMs
• Domyn-Small: A European 10B Reasoning Language Model
• Kimi Linear: An Expressive, Efficient Attention Architecture
• EntropyLong: Effective Long-Context Training via Predictive Uncertainty
• On the Existence and Behavior of Secondary Attention Sinks
• Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM
• Towards Long-window Anchoring in Vision-Language Model Distillation
• Video Detective: Seek Critical Clues Recurrently to Answer Question from Long Videos
• Decoupling Positional and Symbolic Attention Behavior in Transformers
• Spatia: Video Generation with Updatable Spatial Memory
• Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs
• Learning to Focus: Focal Attention for Selective and Scalable Transformers
• Search Is Not Retrieval: Decoupling Semantic Matching from Contextual Assembly in RAG
• Adaptive Context Length Optimization with Low-Frequency Truncation for Multi-Agent Reinforcement Learning
• COOPERA: Continual Open-Ended Human-Robot Assistance
• A Survey on LLM Mid-Training
• Agentic Reinforcement Learning for Real-World Code Repair
• Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity
• Investigating LLM Capabilities on Long Context Comprehension for Medical Question Answering
• Predicting Task Performance with Context-aware Scaling Laws
• Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries
• On Pretraining for Project-Level Code Completion
• xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
• Mid-Training of Large Language Models: A Survey
• Efficient Prediction of Pass@k Scaling in Large Language Models
• Evaluating the Robustness of Chinchilla Compute-Optimal Scaling
• Zeppelin: Balancing Variable-length Workloads in Data Parallel Large Model Training
• Genome modeling and design across all domains of life with Evo 2
• LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMs
• Positional Encoding via Token-Aware Phase Attention
• SciGPT: A Large Language Model for Scientific Literature Understanding and Knowledge Discovery
• Video Parallel Scaling: Aggregating Diverse Frame Subsets for VideoLLMs
• MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining
• HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models
• PLaMo 2 Technical Report
• ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning
• StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding
• Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimization
• A learning-driven automatic planning framework for proton PBS treatments of H&N cancers
• Trainable Dynamic Mask Sparse Attention
• Modality Agnostic Efficient Long Range Encoder
• GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture
• Contextual feature extraction hierarchies converge in large language models and the brain
• Compute Requirements for Algorithmic Innovation in Frontier AI Models
• Agentic Large Language Models for Conceptual Systems Engineering and Design
• MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent
• From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation
• Long-Context Generalization with Sparse Attention
• LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs
• SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models
• Task-Oriented Multimodal Token Transmission in Resource-Constrained Multiuser Networks
• Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenders
• Curse of High Dimensionality Issue in Transformer for Long-context Modeling
• MiniLongBench: The Low-cost Long Context Understanding Benchmark for Large Language Models
• Rotary Masked Autoencoders are Versatile Learners
• Monocle: Hybrid Local-Global In-Context Evaluation for Long-Text Generation with Uncertainty-Based Active Learning
• What Really Matters in Many-Shot Attacks? An Empirical Study of Long-Context Vulnerabilities in LLMs
• QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning
• Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning
• LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
• Scale-invariant Attention
• CoRank: LLM-Based Compact Reranking with Document Features for Scientific Retrieval
• VSA: Faster Video Diffusion with Trainable Sparse Attention
• SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization
• CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capability
• Qwen3 Technical Report
• Classifier Context Rot: Monitor Performance Degrades with Context Length
• AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese
• Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation
• G²-Reader: Dual Evolving Graphs for Multimodal Document QA
• LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
• Compass-V2 Technical Report
• Trillion 7B Technical Report
• One-Minute Video Generation with Test-Time Training

Discussions

• Llama 2 Long [hn, 134 points, 31 comments]
• 📝 Effective Long-Context Scaling of Foundation Models 📚 "Performs a series of long-context LLMs that support effective context windows of up to 32,768 tokens through continual pretraining from Llama [mastodon, 0 points, 0 comments]

Related

• How to Train Long-Context Language Models (Effectively)
• From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
• Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
• LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models
• Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation