Random Bias Initialization Improves Quantized Training

2019/09/30 by Li, Xinlin, Nia, Vahid Partovi
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural and Evolutionary Computing (cs.NE)

paper · doi:10.48550/arxiv.1909.13446

Abstract

Binary neural networks improve computationally efficiency of deep models with a large margin. However, there is still a performance gap between a successful full-precision training and binary training. We bring some insights about why this accuracy drop exists and call for a better understanding of binary network geometry. We start with analyzing full-precision neural networks with ReLU activation and compare it with its binarized version. This comparison suggests to initialize networks with random bias, a counter-intuitive remedy.

Related

• When Bias Helps Learning: Bridging Initial Prejudice and Trainability
• Effects of Initialization Biases on Deep Neural Network Training Dynamics
• Neural Redshift: Random Networks are not Random Functions
• Finding Everything within Random Binary Networks
• On the Impact of the Activation Function on Deep Neural Networks\n Training