2014/11/03 by Charles K. Fisher, Pankaj Mehta, Fisher, Charles K. +1
Biochemistry, Genetics and Molecular Biology · Computer Science · #FOS: Computer and information sciences #FOS: Physical sciences #Gaussian Processes and Bayesian Inference #Gene expression and cancer classification #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Statistical Mechanics (cond-mat.stat-mech)
paper · pdf · doi:10.48550/arxiv.1411.0591
openalex publication_date 2014/11/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Identifying small subsets of features that are relevant for prediction and/or\nclassification tasks is a central problem in machine learning and statistics.\nThe feature selection task is especially important, and computationally\ndifficult, for modern datasets where the number of features can be comparable\nto, or even exceed, the number of samples. Here, we show that feature selection\nwith Bayesian inference takes a universal form and reduces to calculating the\nmagnetizations of an Ising model, under some mild conditions. Our results\nexploit the observation that the evidence takes a universal form for\nstrongly-regularizing priors --- priors that have a large effect on the\nposterior probability even in the infinite data limit. We derive explicit\nexpressions for feature selection for generalized linear models, a large class\nof statistical techniques that include linear and logistic regression. We\nillustrate the power of our approach by analyzing feature selection in a\nlogistic regression-based classifier trained to distinguish between the letters\nB and D in the notMNIST dataset.\n