An Information Bottleneck Problem with Rényi's Entropy

2021/01/29 by Jian-Jia Weng, Fady Alajaji, Weng, Jian-Jia +3
Computer Science · Engineering · Mathematics · #Advanced Data Storage Technologies #Advanced Memory and Neural Computing #FOS: Computer and information sciences #Information Theory (cs.IT) #Wireless Communication Security Techniques #cs.IT #math.IT

paper · pdf · doi:10.48550/arxiv.2101.12564

5 pages, 4 figures, submitted to ISIT 2021

arxiv created 2021/01/29 · openalex publication_date 2021/01/29 · arxiv updated 2021/02/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper considers an information bottleneck problem with the objective of obtaining a most informative representation of a hidden feature subject to a Rényi entropy complexity constraint. The optimal bottleneck trade-off between relevance (measured via Shannon's mutual information) and Rényi entropy cost is defined and an iterative algorithm for finding approximate solutions is provided. We also derive an operational characterization for the optimal trade-off by demonstrating that the optimal Rényi entropy-relevance trade-off is achievable by a simple time-sharing scalar coding scheme and that no coding scheme can provide better performance. Two examples where the optimal Shannon entropy-relevance trade-off can be exactly determined are further given.

Citations

• The information bottleneck method
• Hard Clusters Maximize Mutual Information
• Opening the Black Box of Deep Neural Networks via Information

Related

• On the Relevance-Complexity Region of Scalable Information Bottleneck
• The deterministic information bottleneck
• Practical Estimation of Renyi Entropy
• The Multi-layer Information Bottleneck Problem
• Robust and Fast Measure of Information via Low-rank Representation