Fisher Information and Mutual Information Constraints

2021/02/11 by Barnes, Leighton Pate, Ozgur, Ayfer
#FOS: Computer and information sciences #FOS: Mathematics #Information Theory (cs.IT) #Statistics Theory (math.ST)

paper · doi:10.48550/arxiv.2102.05802

Abstract

We consider the processing of statistical samples X∼ P_θ by a channel p(y|x), and characterize how the statistical information from the samples for estimating the parameter θ∈ℝ^d can scale with the mutual information or capacity of the channel. We show that if the statistical model has a sub-Gaussian score function, then the trace of the Fisher information matrix for estimating θ from Y can scale at most linearly with the mutual information between X and Y. We apply this result to obtain minimax lower bounds in distributed statistical estimation problems, and obtain a tight preconstant for Gaussian mean estimation. We then show how our Fisher information bound can also imply mutual information or Jensen-Shannon divergence based distributed strong data processing inequalities.

Related

• Certain Relations between Mutual Information and Fidelity of Statistical Estimation
• Lower Bounds for Learning Distributions under Communication Constraints via Fisher Information
• On Inequalities Relating the Characteristic Function and Fisher Information
• Converses for distributed estimation via strong data processing inequalities
• Unified lower bounds for interactive high-dimensional estimation under information constraints