Scaling Laws of RoPE-based Extrapolation

2023/10/08 by Xiaoran Liu, Hang Yan, Liu, Xiaoran +9 · 23 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2310.05209

openalex publication_date 2023/10/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The extrapolation capability of Large Language Models (LLMs) based on Rotary Position Embedding is currently a topic of considerable interest. The mainstream approach to addressing extrapolation with LLMs involves modifying RoPE by replacing 10000, the rotary base of θ_n=10000^-2n/d in the original RoPE, with a larger value and providing longer fine-tuning text. In this work, we first observe that fine-tuning a RoPE-based LLM with either a smaller or larger base in pre-training context length could significantly enhance its extrapolation performance. After that, we propose \textbfScaling Laws of RoPE-based Extrapolation, a unified framework from the periodic perspective, to describe the relationship between the extrapolation performance and base value as well as tuning context length. In this process, we also explain the origin of the RoPE-based extrapolation issue by \textbfcritical dimension for extrapolation. Besides these observations and analyses, we achieve extrapolation up to 1 million context length within only 16K training length on LLaMA2 7B and 13B.

Cited by

• Decoupling Positional and Symbolic Attention Behavior in Transformers
• Radiology Report Generation with Layer-Wise Anatomical Attention
• Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs
• BiSparse-AAS: Bilinear Sparse Attention and Adaptive Spans Framework for Scalable and Efficient Text Summarization
• FullPart: Generating each 3D Part at Full Resolution
• SEGA: A Stepwise Evolution Paradigm for Content-Aware Layout Generation with Design Prior
• On Pretraining for Project-Level Code Completion
• Head-wise Adaptive Rotary Positional Encoding for Fine-Grained Image Generation
• Improving GUI Grounding with Explicit Position-to-Coordinate Mapping
• Hyperparameter Loss Surfaces Are Simple Near their Optima
• Mamba Modulation: On the Length Generalization of Mamba
• Positional Encoding via Token-Aware Phase Attention
• Video-LevelGauge: Investigating Contextual Positional Bias in Large Video Language Models
• Flora: Effortless Context Construction to Arbitrary Length and Scale
• Technical Report of TeleChat2, TeleChat2.5 and T1
• MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent
• LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs
• Curse of High Dimensionality Issue in Transformer for Long-context Modeling
• PaTH Attention: Position Encoding via Accumulating Householder Transformations
• Scale-invariant Attention
• Mechanistic Interpretability of GPT-like Models on Summarization Tasks
• Scaling Context, Not Parameters: Training a Compact 7B Language Model for Efficient Long-Context Processing
• SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling
• SmolVLM: Redefining small and efficient multimodal models

Related

• Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation
• Base of RoPE Bounds Context Length
• Rethinking RoPE Scaling in Quantized LLM: Theory, Outlier, and Channel-Band Analysis with Weight Rescaling
• Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective
• Extending Context Window of Large Language Models from a Distributional Perspective