Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization

2019/06/05 by Koen Helwegen, Helwegen, Koen, James Widdicombe +9 · 2 citations
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.1906.02107

openalex publication_date 2019/06/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Optimization of Binarized Neural Networks (BNNs) currently relies on real-valued latent weights to accumulate small update steps. In this paper, we argue that these latent weights cannot be treated analogously to weights in real-valued networks. Instead their main role is to provide inertia during training. We interpret current methods in terms of inertia and provide novel insights into the optimization of BNNs. We subsequently introduce the first optimizer specifically designed for BNNs, Binary Optimizer (Bop), and demonstrate its performance on CIFAR-10 and ImageNet. Together, the redefinition of latent weights as inertia and the introduction of Bop enable a better understanding of BNN optimization and open up the way for further improvements in training methodologies for BNNs. Code is available at: https://github.com/plumerai/rethinking-bnn-optimization

Citations

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Deep Learning with Limited Numerical Precision
• ImageNet Large Scale Visual Recognition Challenge
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Training Quantized Nets: A Deeper Understanding
• Model compression via distillation and quantization
• Probabilistic Binary Neural Networks
• Deep Residual Learning for Image Recognition
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Non-invasive profiling of the tumour microenvironment with spatial ecotypes
• Bi-VLM: Pushing Ultra-Low Precision Post-Training Quantization Boundaries in Vision-Language Models

Related

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Binarized Neural Networks
• How Do Adam and Training Strategies Help BNNs Optimization?
• ReCU: Reviving the Dead Weights in Binary Neural Networks
• OvSW: Overcoming Silent Weights for Accurate Binary Neural Networks