QwenLong-CPRS: Towards ∞-LLMs with Dynamic Context Optimization

2025/05/23 by Shen, Weizhou, Li, Chenliang, Wan, Fanqi +12 · 1 citation
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2505.18092

Abstract

This technical report presents QwenLong-CPRS, a context compression framework designed for explicit long-context optimization, addressing prohibitive computation overhead during the prefill stage and the "lost in the middle" performance degradation of large language models (LLMs) during long sequence processing. Implemented through a novel dynamic context optimization mechanism, QwenLong-CPRS enables multi-granularity context compression guided by natural language instructions, achieving both efficiency gains and improved performance. Evolved from the Qwen architecture series, QwenLong-CPRS introduces four key innovations: (1) Natural language-guided dynamic optimization, (2) Bidirectional reasoning layers for enhanced boundary awareness, (3) Token critic mechanisms with language modeling heads, and (4) Window-parallel inference. Comprehensive evaluations across five benchmarks (4K-2M word contexts) demonstrate QwenLong-CPRS's threefold effectiveness: (1) Consistent superiority over other context management methods like RAG and sparse attention in both accuracy and efficiency. (2) Architecture-agnostic integration with all flagship LLMs, including GPT-4o, Gemini2.0-pro, Claude3.7-sonnet, DeepSeek-v3, and Qwen2.5-max, achieves 21.59× context compression alongside 19.15-point average performance gains; (3) Deployed with Qwen2.5-32B-Instruct, QwenLong-CPRS surpasses leading proprietary LLMs by 4.85 and 10.88 points on Ruler-128K and InfiniteBench, establishing new SOTA performance.

Citations

• Thus Spake Long-Context Large Language Model
• MoBA: Mixture of Block Attention for Long-Context LLMs
• Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
• LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
• Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
• LongAlign: A Recipe for Long Context Alignment of Large Language Models
• Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labeling
• DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
• Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• YaRN: Efficient Context Window Extension of Large Language Models
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
• Towards General Text Embeddings with Multi-stage Contrastive Learning
• Lost in the Middle: How Language Models Use Long Contexts
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• MuSiQue: Multihop Questions via Single-hop Question Composition
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Know What You Don't Know: Unanswerable Questions for SQuAD
• DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications
• Get To The Point: Summarization with Pointer-Generator Networks
• SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
• Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

Cited by

• ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RL

Related

• In-Context Former: Lightning-fast Compressing Context for Large Language Model
• Sentence-Anchored Gist Compression for Long-Context LLMs
• CompLLM: Compression for Long Context Q&A
• LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
• Qwen2.5-1M Technical Report