Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining

2023/02/05 by Zekun Qi, Runpei Dong, Qi, Zekun +11 · 39 citations
Computer Science · Engineering · #3D Shape Modeling and Analysis #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Human Pose and Action Recognition

paper · pdf · doi:10.48550/arxiv.2302.02318

openalex publication_date 2023/02/05 · openalex created_date 2023/02/09 · openalex updated_date 2026/07/28

Abstract

Mainstream 3D representation learning approaches are built upon contrastive or generative modeling pretext tasks, where great improvements in performance on various downstream tasks have been achieved. However, we find these two paradigms have different characteristics: (i) contrastive models are data-hungry that suffer from a representation over-fitting issue; (ii) generative models have a data filling issue that shows inferior data scaling capacity compared to contrastive models. This motivates us to learn 3D representations by sharing the merits of both paradigms, which is non-trivial due to the pattern difference between the two paradigms. In this paper, we propose Contrast with Reconstruct (ReCon) that unifies these two paradigms. ReCon is trained to learn from both generative modeling teachers and single/cross-modal contrastive teachers through ensemble distillation, where the generative student guides the contrastive student. An encoder-decoder style ReCon-block is proposed that transfers knowledge through cross attention with stop-gradient, which avoids pretraining over-fitting and pattern difference issues. ReCon achieves a new state-of-the-art in 3D representation learning, e.g., 91.26% accuracy on ScanObjectNN. Codes have been released at https://github.com/qizekun/ReCon.

Cited by

• Utonia: Toward One Encoder for All Point Clouds
• Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding
• Dual-Branch Center-Surrounding Contrast: Rethinking Contrastive Learning for 3D Point Clouds
• PointDico: Contrastive 3D Representation Learning Guided by Diffusion Models
• CrossJEPA: Cross-Modal Joint-Embedding Predictive Architecture for Efficient 3D Representation Learning from 2D Images
• CASL: Curvature-Augmented Self-supervised Learning for 3D Anomaly Detection
• GFT: Graph Feature Tuning for Efficient Point Cloud Analysis
• URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model
• DAP-MAE: Domain-Adaptive Point Cloud Masked Autoencoder for Effective Cross-Domain Learning
• Reasoning in Space via Grounding in the World
• Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation
• Avi: Action from Volumetric Inference
• Desensitizing for Improving Corruption Robustness in Point Cloud Classification through Adversarial Training
• Point-RTD: Replaced Token Denoising for Pretraining Transformer Models on Point Clouds
• Towards More Diverse and Challenging Pre-training for Point Cloud Learning: Self-Supervised Cross Reconstruction with Decoupled Views
• Adaptive Point-Prompt Tuning: Fine-Tuning Heterogeneous Foundation Models for 3D Point Cloud Analysis
• Masked Clustering Prediction for Unsupervised Point Cloud Pre-training
• Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning
• UniPre3D: Unified Pre-training of 3D Point Cloud Models with Cross-Modal Gaussian Splatting
• TrackAny3D: Transferring Pretrained 3D Models for Category-unified 3D Point Cloud Tracking
• HydraMamba: Multi-Head State Space Model for Global Point Cloud Learning
• UPP: Unified Point-Level Prompting for Robust Point Cloud Analysis
• PointLAMA: Latent Attention meets Mamba for Efficient Point Cloud Pretraining
• MS-DGCNN++: Multi-Scale Dynamic Graph Convolution with Scale-Dependent Normalization for Robust LiDAR Tree Species Classification
• Harnessing Text-to-Image Diffusion Models for Point Cloud Self-Supervised Learning
• Parameter-Efficient Fine-Tuning for Spiking Point Cloud Models
• DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge
• DexVLG: Dexterous Vision-Language-Grasp Model at Scale
• Exploring Semantic Masked Autoencoder for Self-supervised Point Cloud Understanding
• Asymmetric Dual Self-Distillation for 3D Self-Supervised Representation Learning
• GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Model
• OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models
• A Cross Branch Fusion-Based Contrastive Learning Framework for Point Cloud Self-supervised Learning
• On Geometry-Enhanced Parameter-Efficient Fine-Tuning for 3D Scene Segmentation
• PMA: Towards Parameter-Efficient Point Cloud Understanding via Point Mamba Adapter
• OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model
• Generative Data Augmentation for Object Point Cloud Segmentation
• Event ActivityNet: A Large-Scale Simulated-Event Benchmark for Untrimmed Action Understanding
• PointLoRA: Low-Rank Adaptation with Token Selection for Point Cloud Learning

Related

• PointDico: Contrastive 3D Representation Learning Guided by Diffusion Models
• Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learning
• Repulsor: Accelerating Generative Modeling with a Contrastive Memory Bank
• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding
• Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?