APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers

2025/04/03 by Zhuguanyu Wu, Jiayi Zhang, Wu, Zhuguanyu +9 · 19 citations
Engineering · #CCD and CMOS Imaging Sensors #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Infrared Target Detection Methodologies

paper · pdf · doi:10.48550/arxiv.2504.02508

openalex publication_date 2025/04/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Vision Transformers (ViTs) have become one of the most commonly used backbones for vision tasks. Despite their remarkable performance, they often suffer significant accuracy drops when quantized for practical deployment, particularly by post-training quantization (PTQ) under ultra-low bits. Recently, reconstruction-based PTQ methods have shown promising performance in quantizing Convolutional Neural Networks (CNNs). However, they fail when applied to ViTs, primarily due to the inaccurate estimation of output importance and the substantial accuracy degradation in quantizing post-GELU activations. To address these issues, we propose APHQ-ViT, a novel PTQ approach based on importance estimation with Average Perturbation Hessian (APH). Specifically, we first thoroughly analyze the current approximation approaches with Hessian loss, and propose an improved average perturbation Hessian loss. To deal with the quantization of the post-GELU activations, we design an MLP Reconstruction (MR) method by replacing the GELU function in MLP with ReLU and reconstructing it by the APH loss on a small unlabeled calibration set. Extensive experiments demonstrate that APHQ-ViT using linear quantizers outperforms existing PTQ methods by substantial margins in 3-bit and 4-bit across different vision tasks. The source code is available at https://github.com/GoatWu/APHQ-ViT.

Citations

• TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models
• DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformers
• AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer
• PTQ4SAM: Post-Training Quantization for Segment Anything
• Instance-Aware Group Quantization for Vision Transformers
• I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantization
• OcTr: Octree-based Transformer for 3D Object Detection
• RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers
• PD-Quant: Post-Training Quantization based on Prediction Difference Metric
• NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformers
• BiViT: Extremely Compressed Binary Vision Transformer
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer
• QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization
• FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer
• PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization
• Segmenter: Transformer for Semantic Segmentation
• CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
• CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
• A Survey of Quantization Methods for Efficient Neural Network Inference
• BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
• Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
• Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
• Training data-efficient image transformers & distillation through attention
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• End-to-End Object Detection with Transformers
• Up or Down? Adaptive Rounding for Post-Training Quantization
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision
• Learned Step Size Quantization
• Collapse of Deep and Narrow Neural Nets
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Cascade R-CNN: Delving into High Quality Object Detection
• The Expressive Power of Neural Networks: A View from the Width
• Attention Is All You Need
• Mask R-CNN
• Less than a Single Pass: Stochastically Controlled Stochastic Gradient Method
• Fast R-CNN
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Distilling the Knowledge in a Neural Network
• ImageNet Large Scale Visual Recognition Challenge
• ImageNet Large Scale Visual Recognition Challenge
• Microsoft COCO: Common Objects in Context
• Towards Accurate Post-Training Quantization of Vision Transformers via Error Reduction
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• On the mathematical foundations of theoretical statistics

Cited by

• TreeQ: Pushing the Quantization Boundary of Diffusion Transformer via Tree-Structured Mixed-Precision Search
• Activation Quantization of Vision Encoders Needs Prefixing Registers
• Low-bit Model Quantization for Deep Neural Networks: A Survey
• Cat: Post-Training Quantization Error Reduction via Cluster-based Affine Transformation
• I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentation

Related

• PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization
• RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers
• MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformer
• IPTQ-ViT: Post-Training Quantization of Non-linear Functions for Integer-only Vision Transformers
• DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformers