Learning from Simulated and Unsupervised Images through Adversarial Training

2016/12/22 by Ashish Shrivastava, Tomas Pfister, Shrivastava, Ashish +10 · 2 voices · 24 citations
Computer Science · #Adversarial system #Annotation #Artificial intelligence #Computer science #Deep learning #Discriminator #Domain Adaptation and Few-Shot Learning #Human Pose and Action Recognition #Image (mathematics) #Key (lock) #Machine learning #Multimodal Machine Learning Applications #Pattern recognition (psychology) #Regularization (linguistics) #Synthetic data #Unsupervised learning #cs.CV #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1612.07828

published in arXiv (Cornell University) (Cornell University) · Accepted at CVPR 2017 for oral presentation

openalex publication_date 2016/12/22 · arxiv created 2017/07/19 · arxiv updated 2017/07/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations. However, learning from synthetic images may not achieve the desired performance due to a gap between synthetic and real image distributions. To reduce this gap, we propose Simulated+Unsupervised (S+U) learning, where the task is to learn a model to improve the realism of a simulator's output using unlabeled real data, while preserving the annotation information from the simulator. We develop a method for S+U learning that uses an adversarial network similar to Generative Adversarial Networks (GANs), but with synthetic images as inputs instead of random vectors. We make several key modifications to the standard GAN algorithm to preserve annotations, avoid artifacts, and stabilize training: (i) a 'self-regularization' term, (ii) a local adversarial loss, and (iii) updating the discriminator using a history of refined images. We show that this enables generation of highly realistic images, which we demonstrate both qualitatively and with a user study. We quantitatively evaluate the generated images by training models for gaze estimation and hand pose estimation. We show a significant improvement over using synthetic images, and achieve state-of-the-art results on the MPIIGaze dataset without any labeled real data.

Cited by

• Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
• Quantifying and Bridging the Fidelity Gap: A Decisive-Feature Approach to Comparing Synthetic and Real Imagery
• TReFT: Taming Rectified Flow Models For One-Step Image Translation
• Simulator and Experience Enhanced Diffusion Model for Comprehensive ECG Generation
• Beyond Real Data: Synthetic Data through the Lens of Regularization
• High-dimensional Analysis of Synthetic Data Selection
• UniPose: Unified Cross-modality Pose Prior Propagation towards RGB-D data for Weakly Supervised 3D Human Pose Estimation
• Metric-Guided Synthetic Image Data Rendering for Deep Learning compatible with Agentic AI
• Adversarial Appearance Learning in Augmented Cityscapes for Pedestrian Recognition in Autonomous Driving
• Label-Free Whole Slide Virtual Multi-Staining Using Dual-Excitation Photon Absorption Remote Sensing Microscopy
• Domain Adaptation via Feature Refinement
• MedShift: Implicit Conditional Transport for X-Ray Domain Adaptation
• Synthetic Image Detection via Spectral Gaps of QC-RBIM Nishimori Bethe-Hessian Operators
• Stable-Sim2Real: Exploring Simulation of Real-Captured 3D Data with Two-Stage Depth Diffusion
• Bringing Balance to Hand Shape Classification: Mitigating Data Imbalance Through Generative Models
• CycleVAR: Repurposing Autoregressive Model for Unsupervised One-Step Image Translation
• Development of MR spectral analysis method robust against static magnetic field inhomogeneity
• Think before You Simulate: Symbolic Reasoning to Orchestrate Neural Computation for Counterfactual Question Answering
• Disrupting Vision-Language Model-Driven Navigation Services via Adversarial Object Fusion
• Towards Generating Realistic Underwater Images
• Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control
• Knitting Robots: A Deep Learning Approach for Reverse-Engineering Fabric Patterns
• Dual variational autoencoders for efficient sim-to-real transfer in low-cost robotic navigation
• Synthetic data [wikipedia]

Discussions

• Learning from Simulated and Unsupervised Images through Adversarial Training [hn, 103 points, 9 comments]
• Love that apple finally contributes to AI science. Step into right direction. https://arxiv.org/abs/1612.07828 [bsky, 0 points, 0 comments]

Related

• Beyond Photo Realism for Domain Adaptation from Synthetic Data
• Unsupervised Domain Adaptation for Learning Eye Gaze from a Million Synthetic Images: An Adversarial Approach
• Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
• Improved Techniques for Training GANs
• From Real to Synthetic and Back: Synthesizing Training Data for Multi-Person Scene Understanding