Learning from Simulated and Unsupervised Images through Adversarial\n Training

2016/12/22 by Ashish Shrivastava, Tomas Pfister, Shrivastava, Ashish +10 · 2 voices · 15 citations
Computer Science · #Domain Adaptation and Few-Shot Learning #Human Pose and Action Recognition #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.1612.07828

Abstract

With recent progress in graphics, it has become more tractable to train\nmodels on synthetic images, potentially avoiding the need for expensive\nannotations. However, learning from synthetic images may not achieve the\ndesired performance due to a gap between synthetic and real image\ndistributions. To reduce this gap, we propose Simulated+Unsupervised (S+U)\nlearning, where the task is to learn a model to improve the realism of a\nsimulator's output using unlabeled real data, while preserving the annotation\ninformation from the simulator. We develop a method for S+U learning that uses\nan adversarial network similar to Generative Adversarial Networks (GANs), but\nwith synthetic images as inputs instead of random vectors. We make several key\nmodifications to the standard GAN algorithm to preserve annotations, avoid\nartifacts, and stabilize training: (i) a 'self-regularization' term, (ii) a\nlocal adversarial loss, and (iii) updating the discriminator using a history of\nrefined images. We show that this enables generation of highly realistic\nimages, which we demonstrate both qualitatively and with a user study. We\nquantitatively evaluate the generated images by training models for gaze\nestimation and hand pose estimation. We show a significant improvement over\nusing synthetic images, and achieve state-of-the-art results on the MPIIGaze\ndataset without any labeled real data.\n

Cited by

• Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
• Quantifying and Bridging the Fidelity Gap: A Decisive-Feature Approach to Comparing Synthetic and Real Imagery
• TReFT: Taming Rectified Flow Models For One-Step Image Translation
• Simulator and Experience Enhanced Diffusion Model for Comprehensive ECG Generation
• Beyond Real Data: Synthetic Data through the Lens of Regularization
• High-dimensional Analysis of Synthetic Data Selection
• UniPose: Unified Cross-modality Pose Prior Propagation towards RGB-D data for Weakly Supervised 3D Human Pose Estimation
• Metric-Guided Synthetic Image Data Rendering for Deep Learning compatible with Agentic AI
• Adversarial Appearance Learning in Augmented Cityscapes for Pedestrian Recognition in Autonomous Driving
• Label-Free Whole Slide Virtual Multi-Staining Using Dual-Excitation Photon Absorption Remote Sensing Microscopy
• Domain Adaptation via Feature Refinement
• MedShift: Implicit Conditional Transport for X-Ray Domain Adaptation
• Synthetic Image Detection via Spectral Gaps of QC-RBIM Nishimori Bethe-Hessian Operators
• Stable-Sim2Real: Exploring Simulation of Real-Captured 3D Data with Two-Stage Depth Diffusion
• Synthetic data [wikipedia]

Discussions

• Learning from Simulated and Unsupervised Images through Adversarial Training [hn, 103 points, 9 comments]
• Love that apple finally contributes to AI science. Step into right direction. https://arxiv.org/abs/1612.07828 [bsky, 0 points, 0 comments]

Related

• Beyond Photo Realism for Domain Adaptation from Synthetic Data
• Unsupervised Domain Adaptation for Learning Eye Gaze from a Million Synthetic Images: An Adversarial Approach
• Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
• Improved Techniques for Training GANs
• From Real to Synthetic and Back: Synthesizing Training Data for Multi-Person Scene Understanding