2023/01/12 by Robert J. Webber, Webber, Robert J., Matthias Morzfeld +1
Earth and Planetary Sciences · Economics, Econometrics and Finance · Environmental Science · #62H10 #65C20 #86-10 #Atmospheric and Oceanic Physics (physics.ao-ph) #FOS: Computer and information sciences #FOS: Mathematics #FOS: Physical sciences #Meteorological Phenomena and Simulations #Methodology (stat.ME) #Numerical Analysis (math.NA) #Soil Geostatistics and Mapping #Spatial and Panel Data Analysis
paper · pdf · doi:10.48550/arxiv.2301.04828
openalex publication_date 2023/01/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01
A major problem in numerical weather prediction (NWP) is the estimation of high-dimensional covariance matrices from a small number of samples. Maximum likelihood estimators cannot provide reliable estimates when the overall dimension is much larger than the number of samples. Fortunately, NWP practitioners have found ingenious ways to boost the accuracy of their covariance estimators by leveraging the assumption that the correlations decay with spatial distance. In this work, Bayesian statistics is used to provide a new justification and analysis of the practical NWP covariance estimators. The Bayesian framework involves manipulating distributions over symmetric positive definite matrices, and it leads to two main findings: (i) the commonly used "hybrid estimator" for the covariance matrix has a naturally Bayesian interpretation; (ii) the very commonly used "Schur product estimator" is not Bayesian, but it can be studied and understood within the Bayesian framework. As practical implications, the Bayesian framework shows how to reduce the amount of tuning required for covariance estimation, and it suggests that efficient covariance estimation should be rooted in understanding and penalizing conditional correlations, rather than correlations.