Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data

2025/10/29 by Deng, Haoran, Lin, Yingyu, Lin, Zhenghao +4
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2510.25804

Abstract

Long-context language models unlock advanced capabilities in reasoning, code generation, and document summarization by leveraging dependencies across extended spans of text. However, a significant portion of readily available long-text data lacks meaningful long-distance dependencies; most spans can be predicted using only local context. Training on such data is inefficient, making careful data selection crucial. Therefore, we introduce LongFilter, a framework for curating training data tailored to long-context pretraining. LongFilter measures the information gain provided by extended context by contrasting model predictions under long-context versus short-context settings, thereby identifying samples where long-range dependencies are essential. Experiments with LLaMA-3-8B, extending its context length from 8K to 64K, show that LongFilter efficiently selects high-quality data and yields substantial improvements on benchmarks such as HELMET, LongBench, and RULER.

Citations

• Qwen3 Technical Report
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
• LongAttn: Selecting Long-context Training Data via Token-level Attention
• ParallelComp: Parallel Long-Context Compressor for Length Extrapolation
• Phi-4 Technical Report
• How to Train Long-Context Language Models (Effectively)
• The Llama 3 Herd of Models
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• LongWanjuan: Towards Systematic Measurement for Long Text Quality
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
• QuRating: Selecting High-Quality Data for Training Language Models
• Data Engineering for Scaling Language Models to 128K Context
• Extending LLMs' Context Window with 100 Samples
• LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning
• PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
• YaRN: Efficient Context Window Extension of Large Language Models
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
• Extending Context Window of Large Language Models via Positional Interpolation
• Textbooks Are All You Need
• A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity
• Unlimiformer: Long-Range Transformers with Unlimited Length Input
• SemDeDup: Data-efficient learning at web-scale through semantic deduplication
• The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
• Data Selection for Language Models via Importance Resampling
• Perplexed by Quality: A Perplexity-based Method for Adult and Harmful Content Detection in Multilingual Heterogeneous Web Data
• Automatic Document Selection for Efficient Encoder Pretraining
• Improving language models by retrieving from trillions of tokens
• Scaling Language Models: Methods, Analysis & Insights from Training Gopher
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data

Related

• Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models
• How to Train Long-Context Language Models (Effectively)
• Effective Long-Context Scaling of Foundation Models
• LongAttn: Selecting Long-context Training Data via Token-level Attention
• LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs