VideoRoPE: What Makes for Good Video Rotary Position Embedding?

2025/02/07 by Xilin Wei, Wei, Xilin, Xiaoran Liu +21 · 19 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Hand Gesture Recognition Systems #Image and Video Stabilization #Video Analysis and Summarization

paper · pdf · doi:10.48550/arxiv.2502.05173

openalex publication_date 2025/02/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While Rotary Position Embedding (RoPE) and its variants are widely adopted for their long-context capabilities, the extension of the 1D RoPE to video, with its complex spatio-temporal structure, remains an open challenge. This work first introduces a comprehensive analysis that identifies four key characteristics essential for the effective adaptation of RoPE to video, which have not been fully considered in prior work. As part of our analysis, we introduce a challenging V-NIAH-D (Visual Needle-In-A-Haystack with Distractors) task, which adds periodic distractors into V-NIAH. The V-NIAH-D task demonstrates that previous RoPE variants, lacking appropriate temporal dimension allocation, are easily misled by distractors. Based on our analysis, we introduce VideoRoPE, with a 3D structure designed to preserve spatio-temporal relationships. VideoRoPE features low-frequency temporal allocation to mitigate periodic oscillations, a diagonal layout to maintain spatial symmetry, and adjustable temporal spacing to decouple temporal and spatial indexing. VideoRoPE consistently surpasses previous RoPE variants, across diverse downstream tasks such as long video retrieval, video understanding, and video hallucination. Our code will be available at \hrefhttps://github.com/Wiselnn570/VideoRoPEhttps://github.com/Wiselnn570/VideoRoPE.

Cited by

• Flowception: Temporally Expansive Flow Matching for Video Generation
• Minimal Clips, Maximum Salience: Long Video Summarization via Key Moment Extraction
• EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs
• ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning
• The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss
• Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs
• PAS: A Training-Free Stabilizer for Temporal Encoding in Video LLMs
• A Circular Argument : Does RoPE need to be Equivariant for Vision?
• Cambrian-S: Towards Spatial Supersensing in Video
• Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders
• STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence
• Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation
• Revisiting Multimodal Positional Encoding in Vision-Language Models
• Extending Audio Context for Long-Form Understanding in Large Audio-Language Models
• From Pixels to Words -- Towards Native Vision-Language Primitives at Scale
• Head-wise Adaptive Rotary Positional Encoding for Fine-Grained Image Generation
• CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning
• SIM-CoT: Supervised Implicit Chain-of-Thought
• An Empirical Study on How Video-LLMs Answer Video Questions

Related

• VRoPE: Rotary Position Embedding for Video Large Language Models
• ReDirector: Creating Any-Length Video Retakes with Rotary Camera Encoding
• HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models
• AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
• DynImg: Key Frames with Visual Prompts are Good Representation for Multi-Modal Video Understanding