Formal Limitations on the Measurement of Mutual Information

2018/11/10 by David McAllester, McAllester, David, Karl Stratos +1 · 24 citations
Computer Science · #Algorithms and Data Compression #Bayesian Methods and Mixture Models #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms

paper · pdf · doi:10.48550/arxiv.1811.04251

openalex publication_date 2018/11/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confidence lower bound on mutual information estimated from N samples cannot be larger than O(ln N ).

Citations

• Learning deep representations by mutual information estimation and maximization
• Efficient Estimation of Mutual Information for Strongly Dependent Variables

Cited by

• An Information Theoretic Perspective on Agentic System Design
• MIST: Mutual Information Estimation Via Supervised Training
• PCA++: How Uniformity Induces Robustness to Background Noise in Contrastive Learning
• FMMI: Flow Matching Mutual Information Estimation
• Personalized Treatment Outcome Prediction from Scarce Data via Dual-Channel Knowledge Distillation and Adaptive Fusion
• Contrastive Predictive Coding Done Right for Mutual Information Estimation
• Do Latent Channels Actually Communicate? A Causal Audit of Latent Multi-Agent LLM
• Neural Mutual Information Estimation with Vector Copulas
• Connecting Jensen-Shannon and Kullback-Leibler Divergences: A New Bound for Representation Learning
• TENDE: Transfer Entropy Neural Diffusion Estimation
• On the Alignment Between Supervised and Self-Supervised Contrastive Learning
• MMG: Mutual Information Estimation via the MMSE Gap in Diffusion
• NMINE: Normalized Mutual Information Neural Estimation
• DIPHINE: Diffusion-based Φ-ID Neural Estimator
• ML-PWS: Estimating the Mutual Information Between Experimental Time Series Using Neural Networks
• Neural Network Classifier as Mutual Information Evaluator
• Fisher information flow in artificial neural networks
• CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
• Deviation Inequalities for Rényi Divergence Estimators via Variational Expression
• Estimating classical mutual information between quantum subsystems with neural networks
• Understanding the Limitations of Variational Mutual Information Estimators
• Let's Measure Information Step-by-Step: LLM-Based Evaluation Beyond Vibes
• Neural Estimation of Information Leakage for Secure Communication System Design
• Concept Probing: Where to Find Human-Defined Concepts (Extended Version)

Related

• A Tight Lower Bound on the Mutual Information of a Binary and an Arbitrary Finite Random Variable in Dependence of the Variational Distance
• Confidence Intervals for the Mutual Information
• On Variational Bounds of Mutual Information
• Investigation of Alternative Measures for Mutual Information
• Convergence of Statistical Estimators via Mutual Information Bounds