Optimal Information Processing and Bayes's Theorem

1988/11/01 by Arnold Zellner · 263 citations
Computer Science · Mathematics · Physics and Astronomy · #Algorithm #Artificial intelligence #Bayes' theorem #Bayesian probability #Computer science #Entropy (arrow of time) #Fisher information #Inference #Information processing #Information theory #Machine learning #Mathematics #Neural Networks and Applications #Prior information #Statistical Mechanics and Entropy #Statistical inference #Statistics

paper · doi:10.1080/00031305.1988.10475585

published in The American Statistician 42(4), 278-280 (Taylor & Francis)

openalex publication_date 1988/11/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

In this article statistical inference is viewed as information processing involving input information and output information. After introducing information measures for the input and output information, an information criterion functional is formulated and optimized to obtain an optimal information processing rule (IPR). For the particular information measures and criterion functional adopted, it is shown that Bayes's theorem is the optimal IPR. This optimal IPR is shown to be 100% efficient in the sense that its use leads to the output information being exactly equal to the given input information. Also, the analysis links Bayes's theorem to maximum-entropy considerations.

Cited by

• Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networks
• Variational Autoencoding of PDE Inverse Problems
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• Detecting Model Misspecification in Bayesian Inverse Problems via Variational Gradient Descent
• Maxitive Donsker-Varadhan Formulation for Possibilistic Variational Inference
• Quantum measurement retrodiction and entropic uncertainty relations
• Distributed Detection via Bayesian Updates and Consensus
• Provable Bayesian Inference via Particle Mirror Descent
• Multi-Output Robust and Conjugate Gaussian Processes
• Generalised Robust Bayes for Joint Inference of Model and Contamination
• Lüders' and quantum Jeffrey's rules as entropic projections
• Scalable Skewed Bayesian Inference for Latent Gaussian Models Using INLA and Variational Bayes
• Variational Inference based on Robust Divergences
• Stochastic Generative Hashing
• Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production
• Rates of Convergence of Generalised Variational Inference Posteriors under Prior Misspecification
• Assigning a value to a power likelihood in a general Bayesian model
• Weakly Supervised Learning of Heterogeneous Concepts in Videos
• Information Geometry of Variational Bayes
• Unification of Maximum Entropy and Bayesian Inference via Plausible Reasoning
• Generalized Bayesian Filtering via Sequential Monte Carlo
• Priors Matter: Addressing Misspecification in Bayesian Deep Q-Learning
• Offline Policy Selection under Uncertainty
• Bayesian Analysis of Inertial Confinement Fusion Experiments at the National Ignition Facility
• Training Binary Neural Networks using the Bayesian Learning Rule
• Prediction-Oriented Subsampling from Data Streams
• Frequentist Consistency of Generalized Variational Inference
• Kernel Bayesian Inference with Posterior Regularization
• Bayesian reasoning versus conventional statistics in High Energy Physics
• Bayesian Generalized Probability Calculus for Density Matrices
• Energy-Inspired Models: Learning with Sampler-Induced Distributions
• General Information Bottleneck Objectives and their Applications to Machine Learning
• Continuum Limit of Posteriors in Graph Bayesian Inverse Problems
• Improving LoRA with Variational Learning
• Knowledge Adaptation as Posterior Correction
• Variational Learning Finds Flatter Solutions at the Edge of Stability
• Position: There Is No Free Bayesian Uncertainty Quantification
• Variational Deep Learning via Implicit Regularization
• Variational Predictive Information Bottleneck
• Robust Variational Autoencoder
• Bayesian Ensembling: Insights from Online Optimization and Empirical Bayes
• Machine Learning using the Variational Predictive Information Bottleneck with a Validation Set
• Bias or Optimality? Disentangling Bayesian Inference and Learning Biases in Human Decision-Making
• Bayesian Heuristics for Group Decisions
• Distributed Learning for Cooperative Inference
• Robust Deep Gaussian Processes
• A Constrained Matrix-Variate Gaussian Process for Transposable Data
• Reference Analysis
• Evolution of Gaussians in the Hellinger-Kantorovich-Boltzmann gradient flow
• Unifying quantum measurement constructions via a relative-entropy minimum change principle
• mirkwood: Fast and Accurate SED Modeling Using Machine Learning
• Uncertainty and computational complexity. [europepmc]
• Generalized Geographically Weighted Regression Model within a Modularized Bayesian Framework. [europepmc]

Related

• Unification of Maximum Entropy and Bayesian Inference via Plausible Reasoning
• Maximum Entropy: The Universal Method for Inference
• Bayesian Inference and the Principle of Maximum Entropy
• Entropy, Information, and the Updating of Probabilities
• Applications of Information Theory: Statistics and Statistical Mechanics