Training without Gradients -- A Filtering Approach

2020/10/10 by I. Yaesh, Yaesh, Isaac, Natan Grinfeld +1
Computer Science · Engineering · #FOS: Mathematics #Fault Detection and Control Systems #Neural Networks and Applications #Optimization and Control (math.OC) #Target Tracking and Data Fusion in Sensor Networks

paper · pdf · doi:10.48550/arxiv.2010.04908

openalex publication_date 2020/10/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A particle filtering approach is suggested for the training of multi-layer neural networks without utilizing gradients calculation. The network weights are considered to be the components of the estimated state-vector of a noise driven linear system, whereas the neural network serves as the measurement function in the estimation problem. A simple example is used to provide a preliminary demonstration of the concept, which remains to be further studied for training deep neural networks.

Related

• A Tutorial on Neural Networks and Gradient-free Training
• A Global Algorithm for Training Multilayer Neural Networks
• Block-Normalized Gradient Method: An Empirical Study for Training Deep Neural Network
• On the Analysis of Trajectories of Gradient Descent in the Optimization of Deep Neural Networks
• Training Neural Networks Without Gradients: A Scalable ADMM Approach