RSO: A Gradient Free Sampling Based Approach For Training Deep Neural Networks

2020/05/12 by Rohun Tripathi, Bharat Singh, Tripathi, Rohun +1 · 4 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Artificial intelligence #Artificial neural network #Backpropagation #Bayesian probability #Computer science #Deep learning #Deep neural networks #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #MNIST database #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Markov chain #Markov chain Monte Carlo #Pattern recognition (psychology) #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2005.05955

published in arXiv (Cornell University) (Cornell University) · Technical Report

arxiv created 2020/05/12 · openalex publication_date 2020/05/12 · arxiv updated 2020/05/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose RSO (random search optimization), a gradient free Markov Chain Monte Carlo search based approach for training deep neural networks. To this end, RSO adds a perturbation to a weight in a deep neural network and tests if it reduces the loss on a mini-batch. If this reduces the loss, the weight is updated, otherwise the existing weight is retained. Surprisingly, we find that repeating this process a few times for each weight is sufficient to train a deep neural network. The number of weight updates for RSO is an order of magnitude lesser when compared to backpropagation with SGD. RSO can make aggressive weight updates in each step as there is no concept of learning rate. The weight update step for individual layers is also not coupled with the magnitude of the loss. RSO is evaluated on classification tasks on MNIST and CIFAR-10 datasets with deep neural networks of 6 to 10 layers where it achieves an accuracy of 99.1% and 81.8% respectively. We also find that after updating the weights just 5 times, the algorithm obtains a classification accuracy of 98% on MNIST.

Citations

• Gradient-based learning applied to document recognition
• Training Products of Experts by Minimizing Contrastive Divergence
• Pruning Filters for Efficient ConvNets
• Neural Architecture Search with Reinforcement Learning
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Learning Overparameterized Neural Networks via Stochastic Gradient\n Descent on Structured Data
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Understanding Generalization through Visualizations

Related

• Training Deep Neural Networks via Direct Loss Minimization
• A Framework for Provably Stable and Consistent Training of Deep Feedforward Networks
• Learning Gradient Descent: Better Generalization and Longer Horizons
• Repeated Random Sampling for Minimizing the Time-to-Accuracy of Learning
• Reducing Bias in Deep Learning Optimization: The RSGDM Approach