Populate-A-Scene: Affordance-Aware Human Video Generation

2025/07/01 by Shan, Mengyi, He, Zecheng, Ma, Haoyu +4
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2507.00334

Abstract

Can a video generation model be repurposed as an interactive world simulator? We explore the affordance perception potential of text-to-video models by teaching them to predict human-environment interaction. Given a scene image and a prompt describing human actions, we fine-tune the model to insert a person into the scene, while ensuring coherent behavior, appearance, harmonization, and scene affordance. Unlike prior work, we infer human affordance for video generation (i.e., where to insert a person and how they should behave) from a single scene image, without explicit conditions like bounding boxes or body poses. An in-depth study of cross-attention heatmaps demonstrates that we can uncover the inherent affordance perception of a pre-trained video model without labeled affordance datasets.

Citations

• Animate Anyone 2: High-Fidelity Character Image Animation with Environment Affordance
• WorldSimBench: Towards Video Generation Models as World Simulators
• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
• InVi: Object Insertion In Videos Using Off-the-Shelf Diffusion Models
• Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance
• Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance
• AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks
• ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation
• VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
• Latte: Latent Diffusion Transformer for Video Generation
• Photorealistic Video Generation with Diffusion Models
• SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Control
• GenZI: Zero-Shot 3D Human-Scene Interaction Generation
• One-Shot Open Affordance Learning with Foundation Models
• Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation
• Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer
• MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model
• Make Pixels Dance: High-Dynamic Video Generation
• Demystifying CLIP Data
• Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack
• DAT++: Spatially Dynamic Vision Transformer with Deformable Attention
• CoTDet: Affordance Knowledge Prompting for Task Driven Object Detection
• Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models
• Putting People in Their Place: Affordance-Aware Human Insertion into Scenes
• Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models
• DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion
• Micrograph segmentations for DDEVD
• Segment Anything
• Affordance Grounding from Demonstration Video to Target Image
• Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
• Structure and Content-Guided Video Synthesis with Diffusion Models
• Scalable Diffusion Models with Transformers
• Latent Video Diffusion Models for High-Fidelity Long Video Generation
• InstructPix2Pix: Learning to Follow Image Editing Instructions
• Flow Matching for Generative Modeling
• Make-A-Video: Text-to-Video Generation without Text-Video Data
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Grounded Language-Image Pre-training
• One-Shot Affordance Detection
• ByT5: Towards a token-free future with pre-trained byte-to-byte models
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields
• Towards Accurate Generative Models of Video: A New Metric & Challenges
• Learning to Act Properly: Predicting and Explaining Affordances from Images
• Binge Watching: Scaling Affordance Learning from Sitcoms
• Attention Is All You Need

Related

• Putting People in Their Place: Affordance-Aware Human Insertion into Scenes
• Binge Watching: Scaling Affordance Learning from Sitcoms
• Implicit Affordance Acquisition via Causal Action-Effect Modeling in the Video Domain
• 2HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videos
• Grounded Human-Object Interaction Hotspots from Video