2015/02/09 by Christian Hennig, Hennig, Christian, Chien‐Ju Lin +1
Computer Science · Medicine · #62F03 #62F40 #62H30 #Advanced Clustering Algorithms Research #Bayesian Methods and Mixture Models #Data-Driven Disease Surveillance #FOS: Computer and information sciences #Methodology (stat.ME)
paper · pdf · doi:10.48550/arxiv.1502.02574
openalex publication_date 2015/02/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
There are two notoriously hard problems in cluster analysis, estimating the number of clusters, and checking whether the population to be clustered is not actually homogeneous. Given a dataset, a clustering method and a cluster validation index, this paper proposes to set up null models that capture structural features of the data that cannot be interpreted as indicating clustering. Artificial datasets are sampled from the null model with parameters estimated from the original dataset. This can be used for testing the null hypothesis of a homogeneous population against a clustering alternative. It can also be used to calibrate the validation index for estimating the number of clusters, by taking into account the expected distribution of the index under the null model for any given number of clusters. The approach is illustrated by three examples, involving various different clustering techniques (partitioning around medoids, hierarchical methods, a Gaussian mixture model), validation indexes (average silhouette width, prediction strength and BIC), and issues such as mixed type data, temporal and spatial autocorrelation.