Curse of Slicing: Why Sliced Mutual Information is a Deceptive Measure of Statistical Dependence

2025/06/04 by А. І. Семененко, Semenenko, Alexander, Ivan Butakov +5 · 1 citation
Computer Science · Physics and Astronomy · #68T07 #94A16 #94A17 #Age of Information Optimization #Anomaly Detection Techniques and Applications #E.4 #FOS: Computer and information sciences #H.1.1 #Information Theory (cs.IT) #Machine Learning (cs.LG) #Statistical Mechanics and Entropy

paper · pdf · doi:10.48550/arxiv.2506.04053

openalex publication_date 2025/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Sliced Mutual Information (SMI) is widely used as a scalable alternative to mutual information for measuring non-linear statistical dependence. Despite its advantages, such as faster convergence, robustness to high dimensionality, and nullification only under statistical independence, we demonstrate that SMI is highly susceptible to data manipulation and exhibits counterintuitive behavior. Through extensive benchmarking and theoretical analysis, we show that SMI saturates easily, fails to detect increases in statistical dependence, prioritizes redundancy over informative content, and in some cases, performs worse than correlation coefficient.

Citations

• Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMax
• Leveraging Superfluous Information in Contrastive Representation Learning
• Slicing Mutual Information Generalization Bounds for Neural Networks
• InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization
• Max-Sliced Mutual Information
• Beyond Normal: On the Evaluation of Mutual Information Estimators
• Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression
• High-Dimensional Smoothed Entropy Estimation via Dimensionality Reduction
• Conditional Independence Testing via Latent Representation Learning
• Information-Theoretic Generalization Bounds for SGLD via Data-Dependent\n Estimates
• On Mutual Information Maximization for Representation Learning
• Learning Representations by Maximizing Mutual Information Across Views
• Convergence of Smoothed Empirical Measures with Applications to Entropy\n Estimation
• Formal Limitations on the Measurement of Mutual Information
• Deep Graph Infomax
• Learning deep representations by mutual information estimation and maximization
• Representation Learning with Contrastive Predictive Coding
• Chaining Mutual Information and Tightening Generalization Bounds
• Opening the Black Box of Deep Neural Networks via Information
• Deep Learning and the Information Bottleneck Principle
• The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]

Cited by

• FMMI: Flow Matching Mutual Information Estimation

Related

• Sliced Mutual Information: A Scalable Measure of Statistical Dependence
• k-Sliced Mutual Information: A Quantitative Study of Scalability with Dimension
• Max-Sliced Mutual Information
• Scalable Mutual Information Estimation using Dependence Graphs
• Unique Information via Dependency Constraints