2020/06/15 by Alex Mourer, Mourer, Alex, Florent Forest +7 · 1 citation
Computer Science · #62H30 (Primary) 68T10 (Secondary) #Advanced Clustering Algorithms Research #Bayesian Methods and Mixture Models #Data Mining Algorithms and Applications #FOS: Computer and information sciences #I.5.3 #Machine Learning (cs.LG) #Machine Learning (stat.ML)
paper · doi:10.48550/arxiv.2006.08530
openalex publication_date 2020/06/15 · openalex created_date 2023/05/21 · openalex updated_date 2026/07/28
Model selection is a major challenge in non-parametric clustering. There is no universally admitted way to evaluate clustering results for the obvious reason that no ground truth is available. The difficulty to find a universal evaluation criterion is a consequence of the ill-defined objective of clustering. In this perspective, clustering stability has emerged as a natural and model-agnostic principle: an algorithm should find stable structures in the data. If data sets are repeatedly sampled from the same underlying distribution, an algorithm should find similar partitions. However, stability alone is not well-suited to determine the number of clusters. For instance, it is unable to detect if the number of clusters is too small. We propose a new principle: a good clustering should be stable, and within each cluster, there should exist no stable partition. This principle leads to a novel clustering validation criterion based on between-cluster and within-cluster stability, overcoming limitations of previous stability-based methods. We empirically demonstrate the effectiveness of our criterion to select the number of clusters and compare it with existing methods. Code is available at https://github.com/FlorentF9/skstab.