2012/04/30 by Gaurav Pandey, Ambedkar Dukkipati · 4 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Mathematics · #Algorithm #Applied mathematics #Artificial intelligence #Bioinformatics and Genomic Networks #Computer science #Differential entropy #Discriminative model #Entropy (arrow of time) #Feature selection #Gene Regulatory Network Analysis #Gene expression and cancer classification #Joint entropy #Joint quantum entropy #Mathematical optimization #Mathematics #Maximum entropy probability distribution #Maximum entropy spectral estimation #Maximum entropy thermodynamics #Minimax #Minimum description length #Model selection #Principle of maximum entropy #Statistics #cs.IT #math.IT
paper · pdf · doi:10.1109/isit.2013.6620481
9 pages, 3 figures, 4 tables, submitted to Uncertainty in Artificial Intelligence
openalex publication_date 2013/07/01 · arxiv created 2013/11/27 · arxiv updated 2013/11/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
In maximum entropy method, one chooses a distribution from a set of distributions that maximizes the Shannon entropy for making inference from incomplete information. There are various ways to specify this set of distributions, the important special case being when this set is described by mean-value constraints of some feature functions. In this case, maximum entropy method fixes an exponential distribution depending on the feature functions that have to be chosen a priori. In this paper, we treat the problem of selecting a maximum entropy model given various feature subsets and their moments, as a model selection problem, and present a minimum description length (MDL) formulation to solve this problem. For this, we derive normalized maximum likelihood (NML) code-length for these models. Furthermore, we show that the minimax entropy method is a special case of maximum entropy model selection, where one assumes that complexity of all the models are equal. We extend our approach to discriminative maximum entropy models. We apply our approach to gene selection problem to select the number of moments for each gene for fixing the model.