Humans in 4D: Reconstructing and Tracking Humans with Transformers

2023/05/31 by Shubham Goel, Goel, Shubham, Georgios Pavlakos +7 · 64 citations
Computer Science · Engineering · #Human Pose and Action Recognition #Video Surveillance and Tracking Methods #Gait Recognition and Analysis

paper · pdf · doi:10.48550/arxiv.2305.20091

Abstract

We present an approach to reconstruct humans and track them over time. At the core of our approach, we propose a fully "transformerized" version of a network for human mesh recovery. This network, HMR 2.0, advances the state of the art and shows the capability to analyze unusual poses that have in the past been difficult to reconstruct from single images. To analyze video, we use 3D reconstructions from HMR 2.0 as input to a tracking system that operates in 3D. This enables us to deal with multiple people and maintain identities through occlusion events. Our complete approach, 4DHumans, achieves state-of-the-art results for tracking people from monocular video. Furthermore, we demonstrate the effectiveness of HMR 2.0 on the downstream task of action recognition, achieving significant improvements over previous pose-based action recognition approaches. Our code and models are available on the project website: https://shubham-goel.github.io/4dhumans/.

Cited by

• mmSimPrior: Learning Simulation Priors for Data-Efficient Real-World Generalizable Radar-Based Human Motion Reconstruction
• World-Coordinate Human Motion Retargeting via SAM 3D Body
• EchoMotion: Unified Human Video and Motion Generation via Dual-Modality Diffusion Transformer
• VAIR: Visual Analytics for Injury Risk Exploration in Sports
• From Camera to World: A Plug-and-Play Module for Human Mesh Transformation
• CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction
• ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body
• CineLOG: A Training Free Approach for Cinematic Long Video Generation
• MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos
• SAM-Body4D: Training-Free 4D Human Body Mesh Recovery from Videos
• Inferring Compositional 4D Scenes without Ever Seeing One
• Age-Inclusive 3D Human Mesh Recovery for Action-Preserving Data Anonymization
• QuaMo: Quaternion Motions for Vision-based 3D Human Kinematics Capture
• UniMo: Unifying 2D Video and 3D Human Motion with an Autoregressive Framework
• Efficient and Scalable Monocular Human-Object Interaction Motion Reconstruction
• SKEL-CF: Coarse-to-Fine Biomechanical Skeleton and Surface Mesh Recovery
• MonoMSK: Monocular 3D Musculoskeletal Dynamics Estimation
• Robust Long-term Test-Time Adaptation for 3D Human Pose Estimation through Motion Discretization
• VersaPants: A Loose-Fitting Textile Capacitive Sensing System for Lower-Body Motion Capture
• BEDLAM2.0: Synthetic Humans and Cameras in Motion
• Skeletons Speak Louder than Text: A Motion-Aware Pretraining Paradigm for Video-Based Person Re-Identification
• Towards Metric-Aware Multi-Person Mesh Recovery by Jointly Optimizing Human Crowd in Camera Space
• HMVLM: Human Motion-Vision-Lanuage Model via MoE LoRA
• Dynamic Avatar-Scene Rendering from Human-centric Context
• SPIDER: Scalable Physics-Informed Dexterous Retargeting
• MagicView: Multi-View Consistent Identity Customization via Priors-Guided In-Context Learning
• Object-Aware 4D Human Motion Generation
• PHUMA: Physically-Grounded Humanoid Locomotion Dataset
• Sketch2PoseNet: Efficient and Generalized Sketch to 3D Human Pose Prediction
• Demeter: A Parametric Model of Crop Plant Morphology from the Real World
• Kineo: Calibration-Free Metric Motion Capture From Sparse RGB Cameras
• Advances in 4D Representation: Geometry, Motion, and Interaction
• PhysHMR: Learning Humanoid Control Policies from Vision for Physically Plausible Human Motion Reconstruction
• Leveraging 2D Priors and SDF Guidance for Dynamic Urban Scene Rendering
• On the Use of Hierarchical Vision Foundation Models for Low-Cost Human Mesh Recovery and Pose Estimation
• FastHMR: Accelerating Human Mesh Recovery via Token and Layer Merging with Diffusion Decoding
• Mesh-Gait: A Unified Framework for Gait Recognition Through Multi-Modal Representation Learning from 2D Silhouettes
• SyncHuman: Synchronizing 2D and 3D Generative Models for Single-view Human Reconstruction
• DexMan: Learning Bimanual Dexterous Manipulation from Human and Generated Videos
• Human3R: Everyone Everywhere All at Once
• LieHMR: Autoregressive Human Mesh Recovery with SO(3) Diffusion
• AJAHR: Amputated Joint Aware 3D Human Mesh Recovery
• NGD: Neural Gradient Based Deformation for Monocular Garment Reconstruction
• FineQuest: Adaptive Knowledge-Assisted Sports Video Understanding via Agent-of-Thoughts Reasoning
• On the Skinning of Gaussian Avatars
• Geometric Neural Distance Fields for Learning Human Motion Priors
• Improving Human Motion Plausibility with Body Momentum
• ScoreHOI: Physically Plausible Reconstruction of Human-Object Interaction via Score-Guided Diffusion
• HAMSt3R: Human-Aware Multi-view Stereo 3D Reconstruction
• Advances and Trends in the 3D Reconstruction of the Shape and Motion of Animals
• DevilSight: Augmenting Monocular Human Avatar Reconstruction through a Virtual Perspective
• PHD: Personalized 3D Human Body Fitting with Point Diffusion
• Deep Sensorimotor Control by Imitating Predictive Models of Human Motion
• Learnable SMPLify: A Neural Solution for Optimization-Free Human Pose Inverse Kinematics
• DyCrowd: Towards Dynamic Crowd Reconstruction from a Large-scene Video
• HumanGenesis: Agent-Based Geometric and Generative Modeling for Synthetic Human Dynamics
• RealisMotion: Decomposed Human Motion Control and Video Generation in the World Space
• Pose-RFT: Enhancing MLLMs for 3D Pose Generation via Hybrid Action Reinforcement Fine-Tuning
• VOccl3D: A Video Benchmark Dataset for 3D Human Pose and Shape Estimation under real Occlusions
• DiffCap: Diffusion-based Real-time Human Motion Capture using Sparse IMUs and a Monocular Camera
• One Model for All: Unified Try-On and Try-Off in Any Pose via LLM-Inspired Bidirectional Tweedie Diffusion
• AniMer+: Unified Pose and Shape Estimation Across Mammalia and Aves via Family-Aware Transformer
• Reconstructing 4D Spatial Intelligence: A Survey
• Learning Efficient and Generalizable Human Representation with Human Gaussian Model

Related

• Tracking People with 3D Representations
• Human3R: Everyone Everywhere All at Once
• SAM-Body4D: Training-Free 4D Human Body Mesh Recovery from Videos
• H4D: Human 4D Modeling by Learning Neural Compositional Representation
• Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot