RL-GAN-Net: A Reinforcement Learning Agent Controlled GAN Network for\n Real-Time Point Cloud Shape Completion

2019/04/28 by Muhammad Sarmad, Sarmad, Muhammad, Hyunjoo J. Lee +3 · 4 citations
Engineering · Computer Science · #3D Shape Modeling and Analysis #Computer Graphics and Visualization Techniques #Optical measurement and interference techniques

paper · pdf · doi:10.48550/arxiv.1904.12304

Abstract

We present RL-GAN-Net, where a reinforcement learning (RL) agent provides\nfast and robust control of a generative adversarial network (GAN). Our\nframework is applied to point cloud shape completion that converts noisy,\npartial point cloud data into a high-fidelity completed shape by controlling\nthe GAN. While a GAN is unstable and hard to train, we circumvent the problem\nby (1) training the GAN on the latent space representation whose dimension is\nreduced compared to the raw point cloud input and (2) using an RL agent to find\nthe correct input to the GAN to generate the latent space representation of the\nshape that best fits the current input of incomplete point cloud. The suggested\npipeline robustly completes point cloud with large missing regions. To the best\nof our knowledge, this is the first attempt to train an RL agent to control the\nGAN, which effectively learns the highly nonlinear mapping from the input noise\nof the GAN to the latent space of point cloud. The RL agent replaces the need\nfor complex optimization and consequently makes our technique real time.\nAdditionally, we demonstrate that our pipelines can be used to enhance the\nclassification accuracy of point cloud with missing data.\n

Citations

• The Earth Mover's Distance as a Metric for Image Retrieval
• ImageNet classification with deep convolutional neural networks
• An Analysis of Deep Neural Network Models for Practical Applications
• Addressing Function Approximation Error in Actor-Critic Methods
• Self-Attention Generative Adversarial Networks
• Hierarchical Object Detection with Deep Reinforcement Learning
• PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Cited by

• RL-AD-Net: Reinforcement Learning Guided Adaptive Displacement in Latent Space for Refined Point Cloud Completion
• CasPoinTr: Point Cloud Completion with Cascaded Networks and Knowledge Distillation
• Point Cloud Based Reinforcement Learning for Sim-to-Real and Partial Observability in Visual Navigation
• Masked Clustering Prediction for Unsupervised Point Cloud Pre-training

Related

• Controlling the Latent Space of GANs through Reinforcement Learning: A Case Study on Task-based Image-to-Image Translation
• Point Cloud GAN
• GAN Based Top-Down View Synthesis in Reinforcement Learning Environments
• Tempered Adversarial Networks
• Enhanced Experience Replay Generation for Efficient Reinforcement Learning