Batch Normalization-Free Fully Integer Quantized Neural Networks via Progressive Tandem Learning

2025/12/18 by Sun, Pengfei, Jiang, Wenyu, Chee, Piew Yoong +2
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #FOS: Electrical engineering #Machine Learning (cs.LG) #Signal Processing (eess.SP) #Stochastic Gradient Optimization Techniques #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2512.16476

openalex publication_date 2025/12/18 · openalex created_date 2025/12/21 · openalex updated_date 2026/07/28

Abstract

Quantised neural networks (QNNs) shrink models and reduce inference energy through low-bit arithmetic, yet most still depend on a running statistics batch normalisation (BN) layer, preventing true integer-only deployment. Prior attempts remove BN by parameter folding or tailored initialisation; while helpful, they rarely recover BN's stability and accuracy and often impose bespoke constraints. We present a BN-free, fully integer QNN trained via a progressive, layer-wise distillation scheme that slots into existing low-bit pipelines. Starting from a pretrained BN-enabled teacher, we use layer-wise targets and progressive compensation to train a student that performs inference exclusively with integer arithmetic and contains no BN operations. On ImageNet with AlexNet, the BN-free model attains competitive Top-1 accuracy under aggressive quantisation. The procedure integrates directly with standard quantisation workflows, enabling end-to-end integer-only inference for resource-constrained settings such as edge and embedded devices.

Citations

• Distil-DCCRN: A Small-footprint DCCRN Leveraging Feature-based Knowledge Distillation in Speech Enhancement
• Adaptive Axonal Delays in feedforward spiking neural networks for accurate spoken word recognition
• Training Spiking Neural Networks with Local Tandem Learning
• Towards Efficient Full 8-bit Integer DNN Online Training on Resource-limited Devices without Batch Normalization
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks
• Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers
• Learned Step Size Quantization
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Model compression via distillation and quantization
• Training and Inference with Integers in Deep Neural Networks
• Quantization and Training of Neural Networks for Efficient\n Integer-Arithmetic-Only Inference
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Distilling the Knowledge in a Neural Network
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• ImageNet classification with deep convolutional neural networks

Related

• Integer-Only Neural Network Quantization Scheme Based on Shift-Batch-Normalization
• Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond
• Scalable Methods for 8-bit Training of Neural Networks
• Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation