Parameter-Efficient CLIP Adaptation for 3D Understanding via Unified Tokenization

2025/05/24 by Guofeng Mei, Mei, Guofeng, Qinfeng Xiao +17 · 1 citation
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #cs.CV

paper · pdf · doi:10.48550/arxiv.2505.18819

14 pages, tokenizer

arxiv created 2026/07/31 · arxiv updated 2026/08/04

Abstract

Vision-language models, such as CLIP, encode rich semantic knowledge through large-scale image-text pretraining. Reusing these models for 3D understanding is highly desirable, because 3D-text pairs and dense point-level annotations are far scarcer and more difficult to obtain than their 2D counterparts. However, CLIP is trained on regular 2D image patches, whereas point clouds are unordered, sparse, and irregular. The primary challenge lies in constructing 3D token sequences whose geometry, position, and local relations can be interpreted by a frozen, pretrained vision transformer. Moreover, coordinate-scale variations across heterogeneous 3D domains make unified training particularly challenging. To bridge these gaps, we propose a parameter-efficient framework that learns a scale-normalized 3D tokenizer, enabling the CLIP visual encoder to be reused for point-cloud understanding. Our tokenizer, named UTok3D, estimates an input-adaptive geometric scale to calibrate sparse voxelization, coordinate normalization, token-center computation, and 3D positional encoding, and serializes the resulting tokens by Hilbert ordering. This shared geometric convention makes joint training feasible across point clouds with substantially different metric scales, including object-level shapes, indoor scenes, and outdoor LiDAR scans. The tokenizer is trained without 3D annotations through self-supervised cross-modal distillation from foundation-model features extracted from multi-view images, combining local superpoint alignment with the proposed Sinkhorn Ranked Contrastive distillation. Experiments on ShapeNetPart, ScanNetV2, S3DIS, SemanticKITTI, and nuScenes show that, when equipped with a scale-consistent and CLIP-interpretable token interface, a frozen CLIP visual backbone can be effectively reused for annotation-free 3D segmentation.

Citations

• Utonia: Toward One Encoder for All Point Clouds
• Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understanding
• Adaptive Point-Prompt Tuning: Fine-Tuning Heterogeneous Foundation Models for 3D Point Cloud Analysis
• Masked Clustering Prediction for Unsupervised Point Cloud Pre-training
• LOSC: LiDAR Open-voc Segmentation Consolidator
• PGOV3D: Open-Vocabulary 3D Semantic Segmentation with Partial-to-Global Curriculum
• High-quality Pseudo-labeling for Point Cloud Segmentation with Scene-level Annotation
• LogoSP: Local-global Grouping of Superpoints for Unsupervised Semantic Segmentation of 3D Point Clouds
• Sonata: Self-Supervised Learning of Reliable Point Representations
• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation
• Is Contrastive Distillation Enough for Learning Comprehensive 3D Representations?
• High-resolution open-vocabulary object 6D pose estimation
• What If We Recaption Billions of Web Images with LLaMA-3?
• Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation
• Better Call SAL: Towards Learning to Segment Anything in Lidar
• PointMamba: A Simple State Space Model for Point Cloud Analysis
• Self-supervised Learning of LiDAR 3D Point Clouds via 2D-3D Neural Calibration
• Point Transformer V3: Simpler, Faster, Stronger
• PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillation
• Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understanding
• 3D Gaussian Splatting for Real-Time Radiance Field Rendering
• 3D Gaussian Splatting for Real-Time Radiance Field Rendering
• PointGPT: Auto-regressively Generative Pre-training from Point Clouds
• GeoMAE: Masked Geometric Target Prediction for Self-supervised Point Cloud Pre-Training
• DINOv2: Learning Robust Visual Features without Supervision
• Micrograph segmentations for DDEVD
• Segment Anything
• PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection
• A Simple Framework for Open-Vocabulary Segmentation and Detection
• CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP
• Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders
• EPCL: Frozen CLIP Transformer is An Efficient Point Cloud Encoder
• GD-MAE: Generative Decoder for MAE Pre-training on LiDAR Point Clouds
• OpenScene: 3D Scene Understanding with Open Vocabularies
• PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning
• Point Transformer V2: Grouped Vector Attention and Partition-based Pooling
• Pix4Point: Image Pretrained Standard Transformers for 3D Point Cloud Understanding
• P2P: Tuning Pre-trained Image Models for Point Cloud Analysis with Point-to-Pixel Prompting
• Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training
• AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition
• Autoregressive 3D Shape Generation via Canonical Mapping
• Visual Prompt Tuning
• Language-driven Semantic Segmentation
• Scaling Open-Vocabulary Image Segmentation with Image-Level Labels
• PointCLIP: Point Cloud Understanding by CLIP
• Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
• Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds
• Point Discriminative Learning for Data-efficient 3D Point Cloud Analysis
• LoRA: Low-Rank Adaptation of Large Language Models
• Emerging Properties in Self-Supervised Vision Transformers
• Learning Transferable Visual Models From Natural Language Supervision
• PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding
• Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Clouds
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks
• nuScenes: A multimodal dataset for autonomous driving
• Parameter-Efficient Transfer Learning for NLP
• 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks
• Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs
• PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• ShapeNet: An Information-Rich 3D Model Repository

Cited by

• Masked Clustering Prediction for Unsupervised Point Cloud Pre-training

Related

• Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understanding
• CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP
• CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition
• CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP
• Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Models