CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTs

2024/07/07 by Akshat Ramachandran, Souvik Kundu, Ramachandran, Akshat +3 · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #FOS: Electrical engineering #Image and Video Processing (eess.IV) #Machine Learning and ELM #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2407.05266

openalex publication_date 2024/07/07 · openalex created_date 2024/07/10 · openalex updated_date 2026/07/28

Abstract

We present CLAMP-ViT, a data-free post-training quantization method for vision transformers (ViTs). We identify the limitations of recent techniques, notably their inability to leverage meaningful inter-patch relationships, leading to the generation of simplistic and semantically vague data, impacting quantization accuracy. CLAMP-ViT employs a two-stage approach, cyclically adapting between data generation and model quantization. Specifically, we incorporate a patch-level contrastive learning scheme to generate richer, semantically meaningful data. Furthermore, we leverage contrastive learning in layer-wise evolutionary search for fixed- and mixed-precision quantization to identify optimal quantization parameters while mitigating the effects of a non-smooth loss landscape. Extensive evaluations across various vision tasks demonstrate the superiority of CLAMP-ViT, with performance improvements of up to 3% in top-1 accuracy for classification, 0.6 mAP for object detection, and 1.5 mIoU for segmentation at similar or better compression ratio over existing alternatives. Code is available at https://github.com/georgia-tech-synergy-lab/CLAMP-ViT.git

Cited by

• IPTQ-ViT: Post-Training Quantization of Non-linear Functions for Integer-only Vision Transformers
• ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models
• Patch Rebirth: Toward Fast and Transferable Model Inversion of Vision Transformers

Related

• CPT-V: A Contrastive Approach to Post-Training Quantization of Vision Transformers
• AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers
• I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantization
• ADFQ-ViT: Activation-Distribution-Friendly Post-Training Quantization for Vision Transformers
• PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformers