FreeAction: Training-Free Techniques for Enhanced Fidelity of Trajectory-to-Video Generation

2025/09/29 by Seungwook Kim, Kim, Seungwook, Seunghyeon Lee +3
Engineering · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Motion and Animation #Robotics (cs.RO)

paper · pdf · doi:10.48550/arxiv.2509.24241

openalex publication_date 2025/09/29 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Generating realistic robot videos from explicit action trajectories is a critical step toward building effective world models and robotics foundation models. We introduce two training-free, inference-time techniques that fully exploit explicit action parameters in diffusion-based robot video generation. Instead of treating action vectors as passive conditioning signals, our methods actively incorporate them to guide both the classifier-free guidance process and the initialization of Gaussian latents. First, action-scaled classifier-free guidance dynamically modulates guidance strength in proportion to action magnitude, enhancing controllability over motion intensity. Second, action-scaled noise truncation adjusts the distribution of initially sampled noise to better align with the desired motion dynamics. Experiments on real robot manipulation datasets demonstrate that these techniques significantly improve action coherence and visual quality across diverse robot environments.

Citations

• ORV: 4D Occupancy-centric Robot Video Generation
• Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control
• EnerVerse-AC: Envisioning Embodied Environments with Action Condition
• Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
• Prediction with Action: Visual Policy Learning via Joint Denoising Process
• iVideoGPT: Interactive VideoGPTs are Scalable World Models
• Multi-view Image Prompted Multi-view Diffusion for Improved 3D Generation
• CorrespondentDream: Enhancing 3D Fidelity of Text-to-3D using Cross-View Correspondences
• Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance
• Learning Interactive Real-World Simulators
• BridgeData V2: A Dataset for Robot Learning at Scale
• Learning Universal Policies via Text-Guided Video Generation
• Scalable Diffusion Models with Transformers
• RT-1: Robotics Transformer for Real-World Control at Scale
• Interactive Language: Talking to Robots in Real Time
• Improving Sample Quality of Diffusion Models Using Self-Attention Guidance
• Classifier-Free Diffusion Guidance
• CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization

Related

• FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models
• AMPLIFY: Actionless Motion Priors for Robot Learning from Videos
• Video2Act: A Dual-System Video Diffusion Policy with Robotic Spatio-Motional Modeling
• DRAW2ACT: Turning Depth-Encoded Trajectories into Robotic Demonstration Videos
• Learning Real-World Action-Video Dynamics with Heterogeneous Masked Autoregression