2018/07/26 by Murat Kocaoglu, Sanjay Shakkottai, Kocaoglu, Murat +7 · 1 citation
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #Bayesian Methods and Mixture Models #Bayesian Modeling and Causal Inference #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Statistical Methods and Inference
paper · pdf · doi:10.48550/arxiv.1807.10399
openalex publication_date 2018/07/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We study the problem of discovering the simplest latent variable that can make two observed discrete variables conditionally independent. The minimum entropy required for such a latent is known as common entropy in information theory. We extend this notion to Renyi common entropy by minimizing the Renyi entropy of the latent variable. To efficiently compute common entropy, we propose an iterative algorithm that can be used to discover the trade-off between the entropy of the latent variable and the conditional mutual information of the observed variables. We show two applications of common entropy in causal inference: First, under the assumption that there are no low-entropy mediators, it can be used to distinguish causation from spurious correlation among almost all joint distributions on simple causal graphs with two observed variables. Second, common entropy can be used to improve constraint-based methods such as PC or FCI algorithms in the small-sample regime, where these methods are known to struggle. We propose a modification to these constraint-based methods to assess if a separating set found by these algorithms is valid using common entropy. We finally evaluate our algorithms on synthetic and real data to establish their performance.