Obstacles to High-Dimensional Particle Filtering

2008/05/30 by Chris Snyder, Thomas Bengtsson, Peter J. Bickel +3 · 707 citations
Earth and Planetary Sciences · Environmental Science · Mathematics · #Applied mathematics #Climate variability and models #Combinatorics #Data assimilation #Dimension (graph theory) #Ensemble Kalman filter #Exponential growth #Extended Kalman filter #Gaussian #Independent and identically distributed random variables #Kalman filter #Mathematical analysis #Mathematics #Meteorological Phenomena and Simulations #Meteorology #Particle filter #Physics #Quantum mechanics #Random variable #State variable #State vector #Statistical physics #Statistics #Tropical and Extratropical Cyclones Research

paper · pdf · doi:10.1175/2008mwr2529.1

published in Monthly Weather Review 136(12), 4629-4640 (American Meteorological Society)

openalex publication_date 2008/05/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Abstract Particle filters are ensemble-based assimilation schemes that, unlike the ensemble Kalman filter, employ a fully nonlinear and non-Gaussian analysis step to compute the probability distribution function (pdf) of a system’s state conditioned on a set of observations. Evidence is provided that the ensemble size required for a successful particle filter scales exponentially with the problem size. For the simple example in which each component of the state vector is independent, Gaussian, and of unit variance and the observations are of each state component separately with independent, Gaussian errors, simulations indicate that the required ensemble size scales exponentially with the state dimension. In this example, the particle filter requires at least 1011 members when applied to a 200-dimensional state. Asymptotic results, following the work of Bengtsson, Bickel, and collaborators, are provided for two cases: one in which each prior state component is independent and identically distributed, and one in which both the prior pdf and the observation errors are Gaussian. The asymptotic theory reveals that, in both cases, the required ensemble size scales exponentially with the variance of the observation log likelihood rather than with the state dimension per se.

Citations

• Deterministic Nonperiodic Flow

Cited by

• Particle Filtering in Geophysical Systems
• Training-Free Bayesian Filtering with Generative Emulators
• TESTING A PARTICLE FILTER TO RECONSTRUCT CLIMATE CHANGES OVER THE PAST CENTURIES
• Stability of Filters for the Navier-Stokes Equation
• On a class of constrained Bayesian filters and their numerical implementation in high-dimensional state-space Markov models
• A New Particle Filter for Target Tracking in MIMO OFDM Integrated Sensing and Communications
• Efficient sequential Bayesian inference for state-space epidemic models using ensemble data assimilation
• Nonasymptotic bounds for suboptimal importance sampling
• An extended Perron-Frobenius operator filter for nonlinear state estimation
• Penalized Ensemble Kalman Filters for High Dimensional Non-linear Systems
• Can local particle filters beat the curse of dimensionality?
• Autoregressive Boltzmann Generators
• Substantial undocumented infection facilitates the rapid dissemination of novel coronavirus (SARS-CoV-2)
• Meta Flow Maps enable scalable reward alignment
• Data Assimilation: A Mathematical Introduction
• A modified particle filter that reduces weight collapse
• IEnSF: Iterative Ensemble Score Filter for Reducing Error in Posterior Score Estimation in Nonlinear Data Assimilation
• Digital Twins: McKean-Pontryagin Control for Partially Observed Physical Twins
• Probabilistic sequential matrix factorization
• Implicit particle filters for data assimilation
• Flow Matching for Efficient and Scalable Data Assimilation
• A Score-based Diffusion Model Approach for Adaptive Learning of Stochastic Partial Differential Equation Solutions
• Iterated Block Particle Filter for High-dimensional Parameter Learning: Beating the Curse of Dimensionality
• Evolutionary global optimization posed as a randomly perturbed martingale problem and applied to parameter recovery of chaotic oscillators
• Multi-Index Sequential Monte Carlo Methods for partially observed Stochastic Partial Differential Equations
• Nudging the particle filter
• A localized particle filter for geophysical data assimilation
• Nested importance sampling for Bayesian inference: error bounds and the role of dimension
• Optimal Transportation Methods in Nonlinear Filtering: The feedback particle filter
• Particle Filtering for Stochastic Navier-Stokes Signal Observed with\n Linear Additive Noise
• A Polynomial Chaos-Based Kalman Filter Approach for Parameter Estimation of Mechanical Systems
• Using Diffusion Models to do Data Assimilation
• An Invitation to Sequential Monte Carlo Samplers
• Conditional Deep Generative Models for Belief State Planning
• Bayesian Experimental Design for Model Discrepancy Calibration: An Auto-Differentiable Ensemble Kalman Inversion Approach
• On a class of constrained particle filters for continuous-discrete state space models
• Conjugate continuous-discrete projection filter via sparse-Grid quadrature
• Accuracy and stability of the continuous-time 3DVAR filter for the Navier–Stokes equation
• Bridging the ensemble Kalman and particle filters
• Dimensional reduction in nonlinear filtering: A homogenization approach
• Ensemble Score Filter for Data Assimilation of Two-Phase Flow Models in Porous Media
• Sequential Filtering Techniques for Simultaneous Tracking and Parameter Estimation
• State Estimation Using Particle Filtering in Adaptive Machine Learning Methods: Integrating Q-Learning and NEAT Algorithms with Noisy Radar Measurements
• A global optimization paradigm based on change of measures. [europepmc]
• A Survey of Recent Advances in Particle Filters and Remaining Challenges for Multitarget Tracking. [europepmc]
• Inference and control of the nosocomial transmission of methicillin-resistant Staphylococcus aureus . [europepmc]
• Fast uncertainty quantification for dynamic flux balance analysis using non-smooth polynomial chaos expansions. [europepmc]
• Substantial undocumented infection facilitates the rapid dissemination of novel coronavirus (SARS-CoV-2). [europepmc]
• Bayesian Update with Importance Sampling: Required Sample Size. [europepmc]
• A hybrid particle-ensemble Kalman filter for problems with medium nonlinearity. [europepmc]
• Penalized ensemble Kalman filters for high dimensional non-linear systems. [europepmc]
• A particle flow filter for high-dimensional system applications. [europepmc]
• A comparison of nonlinear extensions to the ensemble Kalman filter: Gaussian anamorphosis and two-step ensemble filters. [europepmc]
• A Tempered Particle Filter to Enhance the Assimilation of SAR-Derived Flood Extent Maps Into Flood Forecasting Models. [europepmc]
• Ensemble inference of unobserved infections in networks using partial observations. [europepmc]
• Data assimilation and agent-based modelling: towards the incorporation of categorical agent parameters. [europepmc]
• A Martingale-Free Introduction to Conditional Gaussian Nonlinear Systems. [europepmc]
• Efficient sequential Bayesian inference for state-space epidemic models using ensemble data assimilation. [europepmc]

Related

• Ergodicity and Accuracy of Optimal Particle Filters for Bayesian Data Assimilation
• Error Analysis for the Particle Filter: Methods and Theoretical Support
• Can local particle filters beat the curse of dimensionality?
• Accuracy of the Ensemble Kalman Filter in the Near-Linear Setting
• Non-Bayesian particle filters