2018/05/21 by Eldad Haber, Haber, Eldad, Felix Lucka +3 · 2 citations
Computer Science · Engineering · Mathematics · Physics and Astronomy · #65K10 #65M32 #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural and Evolutionary Computing (cs.NE) #Numerical Analysis (math.NA) #Sparse and Compressive Sensing Techniques #Target Tracking and Data Fusion in Sensor Networks #cs.LG #cs.NA #cs.NE #math.NA #msc:65K10 #msc:65M32
paper · pdf · doi:10.48550/arxiv.1805.08034
10 pages, 2 figures
openalex publication_date 2018/05/21 · arxiv created 2018/05/31 · arxiv updated 2018/06/01 · openalex created_date 2022/10/05 · openalex updated_date 2026/07/28
In this work, we present a new derivative-free optimization method and investigate its use for training neural networks. Our method is motivated by the Ensemble Kalman Filter (EnKF), which has been used successfully for solving optimization problems that involve large-scale, highly nonlinear dynamical systems. A key benefit of the EnKF method is that it requires only the evaluation of the forward propagation but not its derivatives. Hence, in the context of neural networks, it alleviates the need for back propagation and reduces the memory consumption dramatically. However, the method is not a pure "black-box" global optimization heuristic as it efficiently utilizes the structure of typical learning problems. Promising first results of the EnKF for training deep neural networks have been presented recently by Kovachki and Stuart. We propose an important modification of the EnKF that enables us to prove convergence of our method to the minimizer of a strongly convex function. Our method also bears similarity with implicit filtering and we demonstrate its potential for minimizing highly oscillatory functions using a simple example. Further, we provide numerical examples that demonstrate the potential of our method for training deep neural networks.