Ensemble Kalman inversion: a derivative-free technique for machine learning tasks

2018/08/10 by Nikola Kovachki, Nikola B. Kovachki, Andrew M. Stuart · 1 voice · 3 citations
Computer Science · Mathematics · #Algorithm #Applied mathematics #Artificial intelligence #Computer science #Derivative (finance) #Extended Kalman filter #Gaussian Processes and Bayesian Inference #Geology #Inversion (geology) #Kalman filter #Machine learning #Mathematics #Neural Networks and Applications #Statistics #Stochastic Gradient Optimization Techniques #acm:49M15 #acm:65K10 #acm:65L09 #acm:68T20 #cs.LG #math.OC #msc:49M15 #msc:65K10 #msc:65L09 #msc:68T20 #stat.ML

paper · pdf · doi:10.1088/1361-6420/ab1c3a

41 pages, 14 figures

arxiv created 2018/08/10 · arxiv published 2018/08/10 · openalex publication_date 2019/04/24 · arxiv updated 2019/09/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Abstract The standard probabilistic perspective on machine learning gives rise to empirical risk-minimization tasks that are frequently solved by stochastic gradient descent (SGD) and variants thereof. We present a formulation of these tasks as classical inverse or filtering problems and, furthermore, we propose an efficient, gradient-free algorithm for finding a solution to these problems using ensemble Kalman inversion (EKI). The method is inherently parallelizable and is applicable to problems with non-differentiable loss functions, for which back-propagation is not possible. Applications of our approach include offline and online supervised learning with deep neural networks, as well as graph-based semi-supervised learning. The essence of the EKI procedure is an ensemble based approximate gradient descent in which derivatives are replaced by differences from within the ensemble. We suggest several modifications to the basic method, derived from empirically successful heuristics developed in the context of SGD. Numerical results demonstrate wide applicability and robustness of the proposed algorithm.

Citations

• Adam: A Method for Stochastic Optimization
• Intriguing properties of neural networks
• FitNets: Hints for Thin Deep Nets
• On the difficulty of training Recurrent Neural Networks
• Distributed optimization of deeply nested systems
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Data Assimilation: A Mathematical Introduction
• Explaining and Harnessing Adversarial Examples
• A Stochastic Approximation Method
• Stochastic Estimation of the Maximum of a Regression Function
• The perceptron: A probabilistic model for information storage and organization in the brain.
• Gradient-based learning applied to document recognition
• Sequence to Sequence Learning with Neural Networks
• A tutorial on spectral clustering
• Fractional Max-Pooling
• Ensemble Kalman methods for inverse problems
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• ImageNet classification with deep convolutional neural networks
• All you need is a good init
• Gradient Descent Converges to Minimizers
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Instance Normalization: The Missing Ingredient for Fast Stylization
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Never look back - A modified EnKF method and its application to the training of neural networks without back propagation
• A Convergence Theory for Deep Learning via Over-Parameterization
• Deep learning
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Ridge Regression: Biased Estimation for Nonorthogonal Problems
• Ridge Regression: Biased Estimation for Nonorthogonal Problems
• Autoregressive Convolutional Neural Networks for Asynchronous Time Series

Cited by

• Likelihood-informed dimension reduction across tempered Bayesian posteriors
• Bayesian Experimental Design for Model Discrepancy Calibration: An Auto-Differentiable Ensemble Kalman Inversion Approach
• Kalman-Langevin dynamics : exponential convergence, particle approximation and numerical approximation

Discussions

• Ensemble Kalman Inversion: Derivative-Free Machine Learning Tasks [hn, 2 points, 0 comments]

Related

• Ensemble Kalman filter for neural network based one-shot inversion
• Iterative Ensemble Kalman Methods: A Unified Perspective with Some New Variants
• Accuracy Boost in Ensemble Kalman Inversion via Ensemble Control Strategies
• Convergence Acceleration of Ensemble Kalman Inversion in Nonlinear Settings
• l_p regularization for ensemble Kalman inversion