2019/10/20 by Fabio Massimo Zennaro, Ke Chen, Zennaro, Fabio Massimo +1
Computer Science · Engineering · Mathematics · #Bayesian Modeling and Causal Inference #FOS: Computer and information sciences #FOS: Electrical engineering #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Signal Processing (eess.SP) #cs.LG #eess.SP #electronic engineering #information engineering #stat.ML
paper · pdf · doi:10.48550/arxiv.1910.08964
8 pages, 5 figures
arxiv created 2019/10/20 · openalex publication_date 2019/10/20 · arxiv updated 2019/10/22 · openalex created_date 2019/10/25 · openalex updated_date 2026/08/01
In this paper we examine a formalization of feature distribution learning (FDL) in information-theoretic terms relying on the analytical approach and on the tools already used in the study of the information bottleneck (IB). It has been conjectured that the behavior of FDL algorithms could be expressed as an optimization problem over two information-theoretic quantities: the mutual information of the data with the learned representations and the entropy of the learned distribution. In particular, such a formulation was offered in order to explain the success of the most prominent FDL algorithm, sparse filtering (SF). This conjecture was, however, left unproven. In this work, we aim at providing preliminary empirical support to this conjecture by performing experiments reminiscent of the work done on deep neural networks in the context of the IB research. Specifically, we borrow the idea of using information planes to analyze the behavior of the SF algorithm and gain insights on its dynamics. A confirmation of the conjecture about the dynamics of FDL may provide solid ground to develop information-theoretic tools to assess the quality of the learning process in FDL, and it may be extended to other unsupervised learning algorithms.