Learning from Synthetic Humans

2017/01/31 by Gül Varol, Javier Romero, Xavier Martin +5 · 965 citations
Computer Science · Engineering · Psychology · #Artificial intelligence #Computer science #Human Pose and Action Recognition #Robot Manipulation and Learning #Social Robot Interaction and HRI #cs.CV

paper · pdf · doi:10.1109/cvpr.2017.492

Appears in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 9 pages

openalex publication_date 2017/07/01 · arxiv created 2018/01/19 · arxiv updated 2018/01/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Estimating human pose, shape, and motion from images and videos are fundamental challenges with many applications. Recent advances in 2D human pose estimation use large amounts of manually-labeled training data for learning convolutional neural networks (CNNs). Such data is time consuming to acquire and difficult to extend. Moreover, manual labeling of 3D pose, depth and motion is impractical. In this work we present SURREAL (Synthetic hUmans foR REAL tasks): a new large-scale dataset with synthetically-generated but realistic images of people rendered from 3D sequences of human motion capture data. We generate more than 6 million frames together with ground truth pose, depth maps, and segmentation masks. We show that CNNs trained on our synthetic dataset allow for accurate human depth estimation and human part segmentation in real RGB images. Our results and the new dataset open up new possibilities for advancing person analysis using cheap and large-scale synthetic data.

Citations

• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video
• Synthesizing Training Images for Boosting Human 3D Pose Estimation
• MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
• Virtual Worlds as Proxy for Multi-Object Tracking Analysis
• Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views
• Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture

Cited by

• Syn4D: A Multiview Synthetic 4D Dataset
• Benchmarking 3D Human Pose Estimation Models under Occlusions
• Leveraging Synthetic Adult Datasets for Unsupervised Infant Pose Estimation

Related

• Image-based Synthesis for Deep 3D Human Pose Estimation
• MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
• Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision
• Moulding Humans: Non-parametric 3D Human Shape Estimation from Single Images
• Sim2real transfer learning for 3D human pose estimation: motion to the rescue