2021/01/13 by Alexander Marx, Marx, Alexander, Lincen Yang +3 · 1 citation
Computer Science · #Applications (stat.AP) #Bayesian Modeling and Causal Inference #Data Stream Mining Techniques #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning and Algorithms #Machine Learning and Data Classification
paper · pdf · doi:10.48550/arxiv.2101.05009
openalex publication_date 2021/01/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Estimating conditional mutual information (CMI) is an essential yet\nchallenging step in many machine learning and data mining tasks. Estimating CMI\nfrom data that contains both discrete and continuous variables, or even\ndiscrete-continuous mixture variables, is a particularly hard problem. In this\npaper, we show that CMI for such mixture variables, defined based on the\nRadon-Nikodym derivate, can be written as a sum of entropies, just like CMI for\npurely discrete or continuous data. Further, we show that CMI can be\nconsistently estimated for discrete-continuous mixture variables by learning an\nadaptive histogram model. In practice, we estimate such a model by iteratively\ndiscretizing the continuous data points in the mixture variables. To evaluate\nthe performance of our estimator, we benchmark it against state-of-the-art CMI\nestimators as well as evaluate it in a causal discovery setting.\n