Video-P2P: Video Editing with Cross-attention Control
2023/03/08 by Shaoteng Liu, Liu, Shaoteng, Yuechen Zhang +7 · 80 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Multimodal Machine Learning Applications #Video Analysis and Summarization
paper · pdf · doi:10.48550/arxiv.2303.04761
openalex publication_date 2023/03/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
This paper presents Video-P2P, a novel framework for real-world video editing with cross-attention control. While attention control has proven effective for image editing with pre-trained image generation models, there are currently no large-scale video generation models publicly available. Video-P2P addresses this limitation by adapting an image generation diffusion model to complete various video editing tasks. Specifically, we propose to first tune a Text-to-Set (T2S) model to complete an approximate inversion and then optimize a shared unconditional embedding to achieve accurate video inversion with a small memory cost. For attention control, we introduce a novel decoupled-guidance strategy, which uses different guidance strategies for the source and target prompts. The optimized unconditional embedding for the source prompt improves reconstruction ability, while an initialized unconditional embedding for the target prompt enhances editability. Incorporating the attention maps of these two branches enables detailed editing. These technical designs enable various text-driven editing applications, including word swap, prompt refinement, and attention re-weighting. Video-P2P works well on real-world videos for generating new characters while optimally preserving their original poses and scenes. It significantly outperforms previous approaches.
Cited by
- FluencyVE: Marrying Temporal-Aware Mamba with Bypass Attention for Video Editing
- Region-Constraint In-Context Generation for Instructional Video Editing
- EasyV2V: A High-quality Instruction-based Video Editing Framework
- VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization
- IC-Effect: Precise and Efficient Video Effects Editing via In-Context Learning
- OmniView: An All-Seeing Diffusion Model for 3D and 4D View Synthesis
- VABench: A Comprehensive Benchmark for Audio-Video Generation
- Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learning
- FlowSteer: Conditioning Flow Field for Consistent Image Restoration
- Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Edits
- EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing
- Bring Your Dreams to Life: Continual Text-to-Video Customization
- Refaçade: Editing Object with Given Reference Texture
- Zero-Shot Video Translation and Editing with Frame Spatial-Temporal Correspondence
- Video4Spatial: Towards Visuospatial Intelligence with Context-Guided Video Generation
- In-Context Sync-LoRA for Portrait Video Editing
- Generative Video Motion Editing with 3D Point Tracks
- InstanceV: Instance-Level Video Generation
- Video Generation Models Are Good Latent Reward Models
- ObjectAlign: Neuro-Symbolic Object Consistency Verification and Correction
- Point-to-Point: Sparse Motion Guidance for Controllable Video Editing
- Generative Photographic Control for Scene-Consistent Video Cinematic Editing
- InstructMix2Mix: Consistent Sparse-View Editing Through Multi-View Model Personalization
- Redundancy-optimized Multi-head Attention Networks for Multi-View Multi-Label Feature Selection
- UniVA: Universal Video Agent towards Open-Source Next-Generation Video Generalist
- FAME: Fairness-aware Attention-modulated Video Editing
- Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset
- In-Context Learning with Unpaired Clips for Instruction-based Video Editing
- VIDMP3: Video Editing by Representing Motion with Pose and Position Priors
- EditCast3D: Single-Frame-Guided 3D Editing with Video Propagation and View Selection
- FreeViS: Training-free Video Stylization with Inconsistent References
- Mono4DEditor: Text-Driven 4D Scene Editing from Monocular Video via Point-Level Localization of Language-Embedded Gaussians
- UniVideo: Unified Understanding, Generation, and Editing for Videos
- Streaming Drag-Oriented Interactive Video Manipulation: Drag Anything, Anytime!
- IMAGEdit: Let Any Subject Transform
- VRWKV-Editor: Reducing quadratic complexity in transformer-based video editing
- Editable Noise Map Inversion: Encoding Target-image into Noise For High-Fidelity Image Manipulation
- VMDiff: Visual Mixing Diffusion for Limitless Cross-Object Synthesis
- Object-AVEdit: An Object-level Audio-Visual Editing Model
- UniTransfer: Video Concept Transfer via Progressive Spatial and Timestep Decomposition
- EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning
- LoRA-Edit: Controllable First-Frame-Guided Video Editing via Mask-Aware LoRA Fine-Tuning
- Text Slider: Efficient and Plug-and-Play Continuous Concept Control for Image/Video Synthesis via LoRA Adapters
- Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animation
- LazyDrag: Enabling Stable Drag-Based Editing on Multi-Modal Diffusion Transformers via Explicit Correspondence
- Ouroboros: Single-step Diffusion Models for Cycle-consistent Forward and Inverse Rendering
- Tinker: Diffusion's Gift to 3D--Multi-View Consistent Editing From Sparse Inputs without Per-Scene Optimization
- Sketch3DVE: Sketch-based 3D-Aware Scene Video Editing
- Story2Board: A Training-Free Approach for Expressive Storyboard Generation
- Yan: Foundational Interactive Video Generation
- Training-Free Text-Guided Color Editing with Multi-Modal Diffusion Transformer
- DreamVE: Unified Instruction-based Image and Video Editing
- Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesis
- VideoGuard: Protecting Video Content from Unauthorized Editing
- RealisVSR: Detail-enhanced Diffusion for Real-World 4K Video Super-Resolution
- Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis
- LiftVSR: Lifting Image Diffusion to Video Super-Resolution via Hybrid Temporal Modeling with Only 4×RTX 4090s
- DiGA3D: Coarse-to-Fine Diffusional Propagation of Geometry and Appearance for Versatile 3D Inpainting
- DFVEdit: Conditional Delta Flow Vector for Zero-shot Video Editing
- Consistent Video Editing as Flow-Driven Image-to-Video Generation
- Causally Steered Diffusion for Automated Video Counterfactual Generation
- iDiT-HOI: Inpainting-based Hand Object Interaction Reenactment via Video Diffusion Transformer
- Good Noise Makes Good Edits: A Training-Free Diffusion-Based Video Editing with Image and Text Prompts
- TV-LiVE: Training-Free, Text-Guided Video Editing via Layer Informed Vitality Exploitation
- Integrating Large Language Models into Text Animation: An Intelligent Editing System with Inline and Chat Interaction
- FADE: Frequency-Aware Diffusion Model Factorization for Video Editing
- FlowDirector: Training-Free Flow Steering for Precise Text-to-Video Editing
- Interactive Video Generation via Domain Adaptation
- MiniMax-Remover: Taming Bad Noise Helps Video Object Removal
- FlowAlign: Trajectory-Regularized, Inversion-Free Flow-based Image Editing
- CoT-Edit: Let CoT Guide Instruction Video Editing
- ChordVideo: One-Step, Training-Free, Temporally Consistent Video Editing via Low-Energy Transport
- Beyond Consistency: Preserving Temporal Structure in Zero-Shot Video Editing
- DAPE: Dual-Stage Parameter-Efficient Fine-Tuning for Consistent Video Editing with Diffusion Models
- FreePCA: Integrating Consistency Information across Long-short Frames in Training-free Long Video Generation via Principal Component Analysis
- We'll Fix it in Post: Improving Text-to-Video Generation with Neuro-Symbolic Feedback
- Visual Prompting for One-shot Controllable Video Editing without Inversion
- Understanding Attention Mechanism in Video Diffusion Models
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation
- RAGME: Retrieval Augmented Video Generation for Enhanced Motion Realism
Related