A Training-Based Mutual Information Lower Bound for Large-Scale Systems

2021/07/30 by Xiangbo Meng, Kang Gao, Meng, Xiangbo +3
Biochemistry, Genetics and Molecular Biology · Computer Science · #FOS: Computer and information sciences #Gene Regulatory Network Analysis #Information Theory (cs.IT) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2108.00034

openalex publication_date 2021/07/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We provide a mutual information lower bound that can be used to analyze the effect of training in models with unknown parameters. For large-scale systems, we show that this bound can be calculated using the difference between two derivatives of a conditional entropy function. The bound does not require explicit estimation of the unknown parameters. We provide a step-by-step process for computing the bound, and provide an example application. A comparison with known classical mutual information bounds is provided.

Related

• Analyzing Training Using Phase Transitions in Entropy---Part I: General Theory
• Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithms
• A Tight Upper Bound on Mutual Information
• The Minimum Information Principle for Discriminative Learning
• Mutual Information Learned Regressor: an Information-theoretic Viewpoint of Training Regression Systems