DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

2016/06/20 by Shuchang Zhou, Yuxin Wu, Zhou, Shuchang +9 · 101 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE)

paper · pdf · doi:10.48550/arxiv.1606.06160

openalex publication_date 2016/06/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose DoReFa-Net, a method to train convolutional neural networks that have low bitwidth weights and activations using low bitwidth parameter gradients. In particular, during backward pass, parameter gradients are stochastically quantized to low bitwidth numbers before being propagated to convolutional layers. As convolutions during forward/backward passes can now operate on low bitwidth weights and activations/gradients respectively, DoReFa-Net can use bit convolution kernels to accelerate both training and inference. Moreover, as bit convolutions can be efficiently implemented on CPU, FPGA, ASIC and GPU, DoReFa-Net opens the way to accelerate training of low bitwidth neural network on these hardware. Our experiments on SVHN and ImageNet datasets prove that DoReFa-Net can achieve comparable prediction accuracy as 32-bit counterparts. For example, a DoReFa-Net derived from AlexNet that has 1-bit weights, 2-bit activations, can be trained from scratch using 6-bit gradients to get 46.1% top-1 accuracy on ImageNet validation set. The DoReFa-Net AlexNet model is released publicly.

Citations

• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Ternary Weight Networks
• Quantized Convolutional Neural Networks for Mobile Devices

Cited by

• Batch Normalization-Free Fully Integer Quantized Neural Networks via Progressive Tandem Learning
• Arithmetic-Intensity-Aware Quantization
• QuantKAN: A Unified Quantization Framework for Kolmogorov Arnold Networks
• BD-Net: Has Depth-Wise Convolution Ever Been Applied in Binary Neural Networks?
• Extreme Model Compression with Structured Sparsity at Low Precision
• HarmoQ: Harmonized Post-Training Quantization for High-Fidelity Image
• Sharpness-aware Quantization for Deep Neural Networks
• A Survey on Methods and Theories of Quantized Neural Networks
• Learning Architectures for Binary Networks
• Any-Precision Deep Neural Networks
• Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
• SiMaN: Sign-to-Magnitude Network Binarization
• A Survey of FPGA-Based Neural Network Accelerator
• CAT: Compression-Aware Training for bandwidth reduction
• SoFAr: Shortcut-based Fractal Architectures for Binary Convolutional Neural Networks
• AQD: Towards Accurate Fully-Quantized Object Detection
• Trajectory Normalized Gradients for Distributed Optimization
• Toolflows for Mapping Convolutional Neural Networks on FPGAs: A Survey and Future Directions
• Widening and Squeezing: Towards Accurate and Efficient QNNs
• Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Sampling
• ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training
• Adaptive Gradient Quantization for Data-Parallel SGD
• SLSNet: Skin lesion segmentation using a lightweight generative\n adversarial network
• BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer
• SOLE: Hardware-Software Co-design of Softmax and LayerNorm for Efficient Transformer Inference
• O3BNN-R: An Out-of-Order Architecture for High-Performance and Regularized BNN Inference
• BinaryBERT: Pushing the Limit of BERT Quantization
• Quantization of Fully Convolutional Networks for Accurate Biomedical Image Segmentation
• Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks
• True 4-Bit Quantized Convolutional Neural Network Training on CPU: Achieving Full-Precision Parity
• LOTION: Smoothing the Optimization Landscape for Quantized Training
• VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
• AMAQ: Adaptive Mixed-bit Activation Quantization for Collaborative Parameter Efficient Fine-tuning
• A Statistical Framework for Low-bitwidth Training of Deep Neural Networks
• Defend Deep Neural Networks Against Adversarial Examples via Fixed and Dynamic Quantized Activation Functions
• Pretraining Large Language Models with NVFP4
• Adaptive Precision Training: Quantify Back Propagation in Neural Networks with Fixed-point Numbers
• SmartExchange: Trading Higher-cost Memory Storage/Access for Lower-cost Computation
• InfiR2: A Comprehensive FP8 Training Recipe for Reasoning-Enhanced Language Models
• A flexible, extensible software framework for model compression based on\n the LC algorithm
• Binary Autoencoder for Mechanistic Interpretability of Large Language Models
• Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizer
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Mirror Descent View for Neural Network Quantization
• Deep Spiking Neural Networks for Large Vocabulary Automatic Speech Recognition
• Learning Convolutional Networks for Content-weighted Image Compression
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• Trained Ternary Quantization
• GhostSR: Learning Ghost Features for Efficient Image Super-Resolution
• Robustness and Transferability of Universal Attacks on Compressed Models
• Bi-VLM: Pushing Ultra-Low Precision Post-Training Quantization Boundaries in Vision-Language Models
• Searching for Accurate Binary Neural Architectures
• FPGA-Based Accelerators of Deep Learning Networks for Learning and Classification: A Review
• Bit-Flip Attack: Crushing Neural Network with Progressive Bit Search
• A Multi-Grid Implicit Neural Representation for Multi-View Videos
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey
• Deep Lookup Network
• Adaptive Federated Learning With Gradient Compression in Uplink NOMA
• BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization
• Quantization Mimic: Towards Very Tiny CNN for Object Detection
• EMPIR: Ensembles of Mixed Precision Deep Networks for Increased Robustness against Adversarial Attacks
• Training and Inference with Integers in Deep Neural Networks
• Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
• BMXNet: An Open-Source Binary Neural Network Implementation Based on MXNet
• BCNN: Binary Complex Neural Network
• Neural Network Distiller: A Python Package For DNN Compression Research
• Data-Augmented Quantization-Aware Knowledge Distillation
• Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
• DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Scheduling
• Exploiting Information Redundancy in Attention Maps for Extreme Quantization of Vision Transformers
• Quantization Meets OOD: Generalizable Quantization-aware Training from a Flatness Perspective
• Ternary Neural Networks with Fine-Grained Quantization
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks
• Systolic Array-based Architecture for Low-Bit Integerized Vision Transformers
• Progressive Element-wise Gradient Estimation for Neural Network Quantization
• APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration
• Blended Coarse Gradient Descent for Full Quantization of Deep Neural Networks
• Deep Learning Inference in Facebook Data Centers: Characterization,\n Performance Optimizations and Hardware Implications
• Mixed Low-precision Deep Learning Inference using Dynamic Fixed Point
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• Towards Fast and Energy-Efficient Binarized Neural Network Inference on FPGA
• End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost
• GDNSQ: Gradual Differentiable Noise Scale Quantization for Low-bit Neural Networks
• ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
• Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm
• Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distribution
• Training Binary Neural Networks with Real-to-Binary Convolutions
• Training DNNs with Hybrid Block Floating Point
• The Fourth State: Signed-Zero Ternary for Stable LLM Quantization (and More)
• DQT: Dynamic Quantization Training via Dequantization-Free Nested Integer Arithmetic
• QuantNet: Learning to Quantize by Learning within Fully Differentiable Framework
• One Model for All Quantization: A Quantized Network Supporting Hot-Swap Bit-Width Adjustment
• Where and How to Enhance: Discovering Bit-Width Contribution for Mixed Precision Quantization
• Learning Content-Weighted Deep Image Compression
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• RA-BNN: Constructing Robust & Accurate Binary Neural Network to Simultaneously Defend Adversarial Bit-Flip Attack and Improve Accuracy
• Training a Binary Weight Object Detector by Knowledge Transfer for Autonomous Driving
• MSQ: Memory-Efficient Bit Sparsification Quantization
• Model compression as constrained optimization, with application to neural nets. Part V: combining compressions

Related

• Towards Effective Low-bitwidth Convolutional Neural Networks
• Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations
• Exploring the Potential of Low-bit Training of Convolutional Neural Networks
• Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks
• Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks