GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds

2022/12/06 by Honghui Yang, Yang, Honghui, Tong He +13 · 8 citations
Computer Science · Engineering · #3D Shape Modeling and Analysis #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human Pose and Action Recognition

paper · pdf · doi:10.48550/arxiv.2212.03010

openalex publication_date 2022/12/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Despite the tremendous progress of Masked Autoencoders (MAE) in developing vision tasks such as image and video, exploring MAE in large-scale 3D point clouds remains challenging due to the inherent irregularity. In contrast to previous 3D MAE frameworks, which either design a complex decoder to infer masked information from maintained regions or adopt sophisticated masking strategies, we instead propose a much simpler paradigm. The core idea is to apply a Generative Decoder for MAE (GD-MAE) to automatically merges the surrounding context to restore the masked geometric knowledge in a hierarchical fusion manner. In doing so, our approach is free from introducing the heuristic design of decoders and enjoys the flexibility of exploring various masking strategies. The corresponding part costs less than 12% latency compared with conventional methods, while achieving better performance. We demonstrate the efficacy of the proposed method on several large-scale benchmarks: Waymo, KITTI, and ONCE. Consistent improvement on downstream detection tasks illustrates strong robustness and generalization capability. Not only our method reveals state-of-the-art results, but remarkably, we achieve comparable accuracy even with 20% of the labeled data on the Waymo dataset. Code will be released at https://github.com/Nightmare-n/GD-MAE.

Cited by

• Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
• DAP-MAE: Domain-Adaptive Point Cloud Masked Autoencoder for Effective Cross-Domain Learning
• Transferable Class Statistics and Multi-scale Feature Approximation for 3D Object Detection
• Masked Clustering Prediction for Unsupervised Point Cloud Pre-training
• Deep Space Weather Model: Long-Range Solar Flare Prediction from Multi-Wavelength Images
• TurboTrain: Towards Efficient and Balanced Multi-Task Learning for Multi-Agent Perception and Prediction
• Enhancing LiDAR Point Features with Foundation Model Priors for 3D Object Detection
• Harnessing Text-to-Image Diffusion Models for Point Cloud Self-Supervised Learning

Related

• MAELi: Masked Autoencoder for Large-Scale LiDAR Point Clouds
• 3D Feature Prediction for Masked-AutoEncoder-Based Point Cloud Pretraining
• Occupancy-MAE: Self-supervised Pre-training Large-scale LiDAR Point Clouds with Masked Occupancy Autoencoders
• Multi-Scale Neighborhood Occupancy Masked Autoencoder for Self-Supervised Learning in LiDAR Point Clouds
• Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training