On Misspecified Error Distributions in Bayesian Functional Clustering: Consequences and Remedies

2025/10/20 by Fumiya Iwashige, Iwashige, Fumiya, Tomoya Wakayama +5
Computer Science · Mathematics · #Bayesian Methods and Mixture Models #Gaussian Processes and Bayesian Inference #Statistical Methods and Bayesian Inference

paper · pdf · doi:10.48550/arxiv.2510.17215

Abstract

Nonparametric Bayesian approaches provide a flexible framework for clustering without pre-specifying the number of groups, yet they are well known to overestimate the number of clusters, especially for functional data. We show that a fundamental cause of this phenomenon lies in misspecification of the error structure: errors are conventionally assumed to be independent across observed points in Bayesian functional models. Through high-dimensional clustering theory, we demonstrate that ignoring the underlying correlation leads to excess clusters regardless of the flexibility of prior distributions. Guided by this theory, we propose incorporating the underlying correlation structures via Gaussian processes and also present its scalable approximation with principled hyperparameter selection. Numerical experiments illustrate that even simple clustering based on Dirichlet processes performs well once error dependence is properly modeled.

Citations

• Clustering functional data with measurement errors: a simulation-based approach
• A Bayesian functional model with multilevel partition priors for group studies in neuroscience
• Functional Data Analysis: An Introduction and Recent Developments
• Functional Data Analysis: An Introduction and Recent Developments
• Bayesian mixture models (in)consistency for the number of clusters
• Review of Clustering Methods for Functional Data
• Fast Convergence on Perfect Classification for Functional Data
• Post-Processed Posteriors for Banded Covariances
• An enriched mixture model for functional clustering
• Optimal Bayesian estimators for latent variable cluster models
• Mixture models with a prior on the number of components
• Concentration Inequalities and Moment Bounds for Sample Covariance Operators
• Inconsistency of Pitman-Yor process mixtures for the number of components
• Hanson-Wright inequality and sub-gaussian concentration
• A simple example of Dirichlet process mixture inconsistency for the\n number of components

Related

• An enriched mixture model for functional clustering
• Effects of Additional Data on Bayesian Clustering
• Flexible Bayesian Nonparametric Product Mixtures for Multi-scale\n Functional Clustering
• Penalized Clustering of Large Scale Functional Data with Multiple Covariates
• An Experimental Comparison of Several Clustering and Initialization Methods