2020/12/02 by Kang Gao, Bertrand M. Hochwald, Gao, Kang +1
Computer Science · Engineering · Physics and Astronomy · #Advanced Thermodynamics and Statistical Mechanics #FOS: Computer and information sciences #Information Theory (cs.IT) #Matrix Theory and Algorithms #Sparse and Compressive Sensing Techniques
paper · pdf · doi:10.48550/arxiv.2012.00970
openalex publication_date 2020/12/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We analyze phase transitions in the conditional entropy of a sequence caused by a change in the conditional variables. Such transitions happen, for example, when training to learn the parameters of a system, since the transition from the training phase to the data phase causes a discontinuous jump in the conditional entropy of the measured system response. For large-scale systems, we present a method of computing a bound on the mutual information obtained with one-shot training, and show that this bound can be calculated using the difference between two derivatives of a conditional entropy. The system model does not require Gaussianity or linearity in the parameters, and does not require worst-case noise approximations or explicit estimation of any unknown parameters. The model applies to a broad range of algorithms and methods in communication, signal processing, and machine learning that employ training as part of their operation.