Rescaling-Aware Training for Efficient Deployment of Deep Learning Models on Full-Integer Hardware

2025/10/13 by Mueller, Lion, Garcia-Ortiz, Alberto, Najafi, Ardalan +2
#FOS: Computer and information sciences #Hardware Architecture (cs.AR) #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2510.11484

Abstract

Integer AI inference significantly reduces computational complexity in embedded systems. Quantization-aware training (QAT) helps mitigate accuracy degradation associated with post-training quantization but still overlooks the impact of integer rescaling during inference, which is a hardware costly operation in integer-only AI inference. This work shows that rescaling cost can be dramatically reduced post-training, by applying a stronger quantization to the rescale multiplicands at no model-quality loss. Furthermore, we introduce Rescale-Aware Training, a fine tuning method for ultra-low bit-width rescaling multiplicands. Experiments show that even with 8x reduced rescaler widths, the full accuracy is preserved through minimal incremental retraining. This enables more energy-efficient and cost-efficient AI inference for resource-constrained embedded systems.

Citations

• eIQ Neutron: Redefining Edge-AI Inference with Integrated NPU and Compiler Innovations
• DenseShift: Towards Accurate and Efficient Low-Bit Power-of-Two Quantization
• Power-of-Two Quantization for Low Bitwidth and Hardware Compliant Neural Networks
• Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
• Fighting Quantization Bias With Bias
• Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

Related

• Training and Inference with Integers in Deep Neural Networks
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• Compute-Optimal Quantization-Aware Training
• A White Paper on Neural Network Quantization
• EasyQuant: Post-training Quantization via Scale Optimization