2024/05/23 by Zeqi Xiao, Xiao, Zeqi, Yifan Zhou +5 · 13 citations
Physics and Astronomy · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Model Reduction and Neural Networks
paper · pdf · doi:10.48550/arxiv.2405.14864
openalex publication_date 2024/05/23 · openalex created_date 2024/05/25 · openalex updated_date 2026/07/28
Video generation primarily aims to model authentic and customized motion across frames, making understanding and controlling the motion a crucial topic. Most diffusion-based studies on video motion focus on motion customization with training-based paradigms, which, however, demands substantial training resources and necessitates retraining for diverse models. Crucially, these approaches do not explore how video diffusion models encode cross-frame motion information in their features, lacking interpretability and transparency in their effectiveness. To answer this question, this paper introduces a novel perspective to understand, localize, and manipulate motion-aware features in video diffusion models. Through analysis using Principal Component Analysis (PCA), our work discloses that robust motion-aware feature already exists in video diffusion models. We present a new MOtion FeaTure (MOFT) by eliminating content correlation information and filtering motion channels. MOFT provides a distinct set of benefits, including the ability to encode comprehensive motion information with clear interpretability, extraction without the need for training, and generalizability across diverse architectures. Leveraging MOFT, we propose a novel training-free video motion control framework. Our method demonstrates competitive performance in generating natural and faithful motion, providing architecture-agnostic insights and applicability in a variety of downstream tasks.