Source-Side Sufficiency for the Information Bottleneck: Exact Reduction and Finite-Block Equivalence

2026/04/30 by Joss Armstrong
Computer Science · Mathematics · #cs.IT #math.IT #stat.ML

paper · pdf

arxiv created 2026/08/03 · arxiv updated 2026/08/05

Abstract

The input side of the Information Bottleneck may contain task-irrelevant variation that still costs rate. We identify this cost exactly. Let T be a source and C a relevance variable. Suppose the deterministic statistic Z = phi(T) satisfies C-Z-T. For any encoder p(X|T), its conditional average over the fibres of phi preserves I(X;C) and lowers the rate by I(X;T|Z). The reverse pullback preserves both coordinates. These maps establish equality of the relevance-rate curves and Lagrangian infima on standard Borel spaces for every tradeoff parameter. They also characterise all attained optima. Every full-source minimiser factors through Z, and every reduced minimiser pulls back to T. When C is finite and distortion is logarithmic loss, replacement of Tⁿ by Zⁿ also preserves the optimal remote distortion at every blocklength and message budget. The operational rate-distortion functions are therefore equal.

Citations

• On the Information Bottleneck Problems: Models, Connections, Applications and Information Theoretic Views
• Generalizing Bottleneck Problems
• Gaussian Lower Bound for the Information Bottleneck Limit
• Emergence of Invariance and Disentanglement in Deep Representations
• Multiterminal Source Coding under Logarithmic Loss
• The information bottleneck method
• Equivalent Comparisons of Experiments
• Deep Variational Information Bottleneck