2025/05/21 by Johannes Kaiser, Kaiser, Johannes, Kristian Schwethelm +5 · 1 citation
Decision Sciences · Mathematics · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Forecasting Techniques and Applications #Information Theory (cs.IT) #Machine Learning (cs.LG) #Statistics Education and Methodologies
paper · pdf · doi:10.48550/arxiv.2505.15303
openalex publication_date 2025/05/21 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28
Accurately estimating the informativeness of individual samples in a dataset is an important objective in deep learning, as it can guide sample selection, which can improve model efficiency and accuracy by removing redundant or potentially harmful samples. We propose Laplace Sample Information (LSI) measure of sample informativeness grounded in information theory widely applicable across model architectures and learning settings. LSI leverages a Bayesian approximation to the weight posterior and the KL divergence to measure the change in the parameter distribution induced by a sample of interest from the dataset. We experimentally show that LSI is effective in ordering the data with respect to typicality, detecting mislabeled samples, measuring class-wise informativeness, and assessing dataset difficulty. We demonstrate these capabilities of LSI on image and text data in supervised and unsupervised settings. Moreover, we show that LSI can be computed efficiently through probes and transfers well to the training of large models.