Patch-wise Mixed-Precision Quantization of Vision Transformer

2023/05/11 by Junrui Xiao, Xiao, Junrui, Zhikai Li +5 · 3 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Advanced Neural Network Applications #Image Enhancement Techniques

paper · pdf · doi:10.48550/arxiv.2305.06559

Abstract

As emerging hardware begins to support mixed bit-width arithmetic computation, mixed-precision quantization is widely used to reduce the complexity of neural networks. However, Vision Transformers (ViTs) require complex self-attention computation to guarantee the learning of powerful feature representations, which makes mixed-precision quantization of ViTs still challenging. In this paper, we propose a novel patch-wise mixed-precision quantization (PMQ) for efficient inference of ViTs. Specifically, we design a lightweight global metric, which is faster than existing methods, to measure the sensitivity of each component in ViTs to quantization errors. Moreover, we also introduce a pareto frontier approach to automatically allocate the optimal bit-precision according to the sensitivity. To further reduce the computational complexity of self-attention in inference stage, we propose a patch-wise module to reallocate bit-width of patches in each layer. Extensive experiments on the ImageNet dataset shows that our method greatly reduces the search cost and facilitates the application of mixed-precision quantization to ViTs.

Cited by

• Exploiting Information Redundancy in Attention Maps for Extreme Quantization of Vision Transformers
• LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation
• MixA-Q: Revisiting Activation Sparsity for Vision Transformers from a Mixed-Precision Quantization Perspective

Related

• MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformer
• LampQ: Towards Accurate Layer-wise Mixed Precision Quantization for Vision Transformers
• FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer
• LRP-QViT: Mixed-Precision Vision Transformer Quantization via Layer-wise Relevance Propagation
• ADFQ-ViT: Activation-Distribution-Friendly Post-Training Quantization for Vision Transformers