A Practical & Unified Notation for Information-Theoretic Quantities in ML

2021/06/22 by Andreas Kirsch, Yarin Gal, Kirsch, Andreas +1
Computer Science · Mathematics · #Artificial intelligence #Bayesian inference #Bayesian probability #Computer science #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Inference #Information theory #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #Machine learning #Mathematics #Mutual information #Notation #Programming language #Set (abstract data type) #Set function #Theoretical computer science #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2106.12062

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2021/06/22 · openalex created_date 2021/07/05 · arxiv created 2021/12/02 · arxiv updated 2021/12/03 · openalex updated_date 2026/08/05

Abstract

A practical notation can convey valuable intuitions and concisely express new ideas. Information theory is of importance to machine learning, but the notation for information-theoretic quantities is sometimes opaque. We propose a practical and unified notation and extend it to include information-theoretic quantities between observed outcomes (events) and random variables. This includes the point-wise mutual information known in NLP and mixed quantities such as specific surprise and specific information in the cognitive sciences and information gain in Bayesian optimal experimental design. We apply our notation to prove a version of Stirling's approximation for binomial coefficients mentioned by MacKa (2003) using new intuitions. We also concisely rederive the evidence lower bound for variational auto-encoders and variational inference in approximate Bayesian neural networks. Furthermore, we apply the notation to a popular information-theoretic acquisition function in Bayesian active learning which selects the most informative (unlabelled) samples to be labelled by an expert and extend this acquisition function to the core-set problem with the goal of selecting the most informative samples given the labels.

Citations

• Bayesian Active Learning for Classification and Preference Learning
• Auto-Encoding Variational Bayes
• On a Measure of the Information Provided by an Experiment
• Gradient-based learning applied to document recognition
• Opening the Black Box of Deep Neural Networks via Information
• Variational Bayesian Optimal Experimental Design
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• A Theory of Usable Information Under Computational Constraints
• Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning

Related

• Information-Theoretic Foundations for Machine Learning
• Unifying Approaches in Active Learning and Active Sampling via Fisher Information and Information-Theoretic Quantities
• Information Theory and its Relation to Machine Learning
• Information, learning and falsification
• A Bayesian Framework for Information-Theoretic Probing