Estimation of Entropy and Mutual Information
2003/05/22 by Liam Paninski · 148 citations
Physics and Astronomy · Engineering · Computer Science · #Statistical Mechanics and Entropy #Control Systems and Identification #Gaussian Processes and Bayesian Inference
paper · doi:10.1162/089976603321780272
Abstract
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This “inconsistency” theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual [Formula: see text] formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if “bias-corrected” estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods. Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.
Citations
Cited by
- Performance Guarantees for Quantum Neural Estimation of Entropies
- Nonparametric Estimation of Joint Entropy via Partitioned Sample-Spacing
- Robust Sensitivity Analysis for Stochastic Systems
- Monte Carlo convergence of rival samplers
- Measuring Grammatical Diversity from Small Corpora: Derivational Entropy Rates, Mean Length of Utterances, and Annotation Invariance
- Estimation of discrete distributions with high probability under χ2-divergence
- The Information-Theoretic Imperative: Compression and the Epistemic Foundations of Intelligence
- Uncertainty Quantification for Regression: A Unified Framework based on kernel scores
- Near-Optimal Learning of Tree-Structured Distributions by Chow-Liu
- Positional information, positional error, and read-out precision in morphogenesis: a mathematical framework
- Steganalysis Feature Selection With Multidimensional Evaluation and Dynamic Threshold Allocation
- Adaptive Estimation of Shannon Entropy
- Motor control by precisely timed spike patterns
- MINERVA: Mutual Information Neural Estimation for Supervised Feature Selection
- Cellular noise and information transmission
- SpatEntropy: Spatial Entropy Measures in R
- Testing Closeness With Unequal Sized Samples
- Understanding Convolutional Neural Networks with Information Theory: An Initial Exploration
- Loss Given Default Prediction Under Measurement-Induced Mixture Distributions: An Information-Theoretic Approach
- Mutual Information guided Visual Contrastive Learning
- Neural Mutual Information Estimation with Vector Copulas
- Neural coding of a natural stimulus ensemble: Uncovering information at sub-millisecond resolution
- Data Amplification: Instance-Optimal Property Estimation
- Information-theoretic analysis of temporal dependence in discrete stochastic processes: Application to precipitation predictability
- Information flow in multilayer perceptrons: an in-depth analysis
- Polynomial methods in statistical inference: theory and practice
- Early Exiting with Ensemble Internal Classifiers
- Internal Planning in Language Models: Characterizing Horizon and Branch Awareness
- On Predictive Information in RNNs
- NMINE: Normalized Mutual Information Neural Estimation
- epiGPTope: A Machine Learning-Based Epitope Generator and Classifier
- Millisecond-scale motor encoding in a cortical vocal area
- When trajectory-based bounds fail: information thermodynamics under noisy feedback
- Estimating Conditional Mutual Information for Discrete-Continuous Mixtures using Multi-Dimensional Adaptive Histograms
- Generative vocal plasticity in chimpanzees
- A Regression Approach to Certain Information Transmission Problems
- Confidence Decision Trees via Online and Active Learning for Streaming (BIG) Data
- Improving the Learning of Multi-column Convolutional Neural Network for Crowd Counting
- Approximate Profile Maximum Likelihood
- Structure-Fair Quantum Circuit Complexity: An Auditable Information-Theoretic Lower Bound
- MEC-Quant: Maximum Entropy Coding for Extremely Low Bit Quantization-Aware Training
- Multi-Fidelity Hybrid Reinforcement Learning via Information Gain Maximization
- Information theory and adaptation
- Estimating Semantic Alphabet Size for LLM Uncertainty Quantification
- Information Flow in Computational Systems
- Improved Information Gain Estimates for Decision Tree Induction
- Mutual Information Surprise: Rethinking Unexpectedness in Autonomous Systems
- The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives
- Rate-Distortion Limits for Multimodal Retrieval: Theory, Optimal Codes, and Finite-Sample Guarantees
- Model-Augmented Estimation of Conditional Mutual Information for Feature Selection
- ML-PWS: Estimating the Mutual Information Between Experimental Time Series Using Neural Networks
- An Information Theoretic Approach Towards Assessing Perceptual Audio Quality using EEG
- Information-Theoretic Lower Bounds for Approximating Monomials via Optimal Quantum Tsallis Entropy Estimation
- Bayesian Entropy Estimation for Countable Discrete Distributions
- Optimal Data Reduction under Information-Theoretic Criteria
- Minimax rates of entropy estimation on large alphabets via best polynomial approximation
- Feature Generation for Robust Semantic Role Labeling
- Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination
- Maximum-Entropy Adversarial Data Augmentation for Improved Generalization and Robustness
- Moderate deviation principle for plug-in estimators of diversity indices on countable alphabets
- A Probabilistic Representation of DNNs: Bridging Mutual Information and Generalization
- Identification of Interaction Clusters Using a Semi-supervised Hierarchical Clustering Method
- Deviation Inequalities for Rényi Divergence Estimators via Variational Expression
- Bias Correction with Jackknife, Bootstrap, and Taylor Series
- Mutual Information as a Figure of Merit for Optical Fiber Systems
- The distribution of information content in English sentences
- Sharp Concentration Inequalities for the Centered Relative Entropy
- Performance Assessment of MIMO-BICM Demodulators based on System Capacity
- Forecastable Component Analysis (ForeCA)
- Nearly Minimax Discrete Distribution Estimation in Kullback-Leibler Divergence with High Probability
- Nested Learning For Multi-Granular Tasks
- Privacy with Estimation Guarantees
- Predictable Drifts in Collective Cultural Attention: Evidence from Nation-Level Library Takeout Data
- Regularized Weighted Chebyshev Approximations for Support Estimation
- Fingerprinting Cryptographic Protocols with Key Exchange using an Entropy Measure
- Ranking by Dependence - A Fair Criteria
- Universal Clustering via Crowdsourcing
- Incorporating Interventional Independence Improves Robustness against Interventional Distribution Shift
- PRI-VAE: Principle-of-Relevant-Information Variational Autoencoders
- Large-scale directed network inference with multivariate transfer entropy and hierarchical statistical testing
- Cross-Modal Contrastive Learning for Text-to-Image Generation
- Canonical Thermodynamics
- SparsityBoost: A New Scoring Function for Learning Bayesian Network Structure
- Estimation of KL Divergence: Optimal Minimax Rate
- Entropy Rate Estimation for Markov Chains with Large State Space
- Transfer entropy for finite data
- Probably Approximately Correct Greedy Maximization with Efficient Bounds on Information Gain for Sensor Selection
- Mutual Information Regularized Identity-aware Facial ExpressionRecognition in Compressed Video
- Markov Blanket Density and Free Energy Minimization
- Online Learning of Noisy Data with Kernels
- A Mutual Information Maximization Perspective of Language Representation Learning
- Disruption of transfer entropy and inter-hemispheric brain functional connectivity in patients with disorder of consciousness
- Beyond Maximum Likelihood: from Theory to Practice
- scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omics
- Accurate Estimation of Mutual Information in High Dimensional Data
- Information theoretic analysis of computational models as a tool to understand the neural basis of behaviors
- Understanding Recurrent Neural Architectures by Analyzing and Synthesizing Long Distance Dependencies in Benchmark Sequential Datasets
- An Empowerment-based Solution to Robotic Manipulation Tasks with Sparse Rewards
- Cross-modal Image Retrieval with Deep Mutual Information Maximization
- INSPECTRE: Privately Estimating the Unseen
- The Harmonic Entropy Estimator: Minimax Optimality and Semiparametric Efficiency for Infinite Alphabets
- Fixing a Broken ELBO
- Causality and Generalizability: Identifiability and Learning Methods
- Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens
- Concentration Inequalities for the Empirical Distribution
- Adversity Index for Clinical Trials: An Inclusive Approach for Analysis of Safety Data
- PEER pressure: Model-to-Model Regularization for Single Source Domain Generalization
- Estimating mutual information in high dimensions via classification error
- Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables
- Improved Sample Upper and Lower Bounds for Trace Estimation of Quantum State Powers
- Closed-Form Information Capacity of Canonical Signaling Models
- Vendi Information Gain: An Alternative To Mutual Information For Science And Machine Learning
- Compression, Regularity, Randomness and Emergent Structure: Rethinking Physical Complexity in the Data-Driven Era
- An Approximation Scheme for Multivariate Information based on Partial Information Decomposition
- Neural Entropic Estimation: A faster path to mutual information estimation
- Horizon-resolved Forecastability of Time Series via Auto Mutual Information
- Hierarchical Low-rank Structure of Parameterized Distributions
- Bias analysis in entropy estimation
- Contextures: Representations from Contexts
- KLIP: localized distribution shift detection via KL-divergence with diffusion priors in Inverse Problems
- Entropy estimates of small data sets
- Studying Maximum Information Leakage Using Karush-Kuhn-Tucker Conditions
- The Algebraic Limits of Polynomial Information Measures
- On directed information theory and Granger causality graphs
- Information based Deep Clustering: An experimental study
- Estimators for Multivariate Information Measures in General Probability Spaces
- Information Theoretic Structured Generative Modeling
- Exploring internal representation of self-supervised networks: few-shot learning abilities and comparison with human semantics and recognition of objects
- Psychological entropy: A framework for understanding uncertainty-related anxiety.
- Learning is Forgetting: LLM Training As Lossy Compression
- On the Non-Specificity of Statistical Measures Used in Script Decipherment
- Bayesian and Quasi-Bayesian Estimators for Mutual Information from Discrete Data
- Estimating Functions of Distributions Defined over Spaces of Unknown Size
- Maximizing the Information Content of Experiments in Systems Biology
- Finding Sequence Features in Tissue-specific Sequences
- Warped phase coherence: An empirical synchronization measure combining phase and amplitude information
- Networks in financial markets based on the mutual information rate
- Usable Information and Evolution of Optimal Representations During Training
- The New Urban Success: How Culture Pays
- Mutual Information Scaling for Tensor Network Machine Learning
- Towards data analysis with diagrammatics
- Limit Theorems for Empirical Rényi Entropy and Divergence with Applications to Molecular Diversity Analysis
- Inferring entropy from structure
- Consistent distribution-free tests of association between univariate random variables
- Near-optimal algorithms for private estimation and sequential testing of collision probability
- Information-theoretic approach to lead-lag effect on financial markets
- CROSSAN: Towards Efficient and Effective Adaptation of Multiple Multimodal Foundation Models for Sequential Recommendation
- Partial transfer entropy on rank vectors
Related