Asymptotic Log-loss of Prequential Maximum Likelihood Codes

2005/02/01 by Peter Grünwald, Peter Grunwald, Grunwald, Peter +2 · 1 citation
Computer Science · Engineering · Mathematics · #Advanced Wireless Communication Techniques #Algorithms and Data Compression #E.4 #Error Correcting Code Techniques #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #cs.IT #cs.LG #math.IT

paper · pdf · doi:10.48550/arxiv.cs/0502004

22 pages, an abstract has been submitted to COLT 2005

arxiv created 2005/02/01 · openalex publication_date 2005/02/01 · arxiv updated 2009/12/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We analyze the Dawid-Rissanen prequential maximum likelihood codes relative to one-parameter exponential family models M. If data are i.i.d. according to an (essentially) arbitrary P, then the redundancy grows at rate c/2 ln n. We show that c=v1/v2, where v1 is the variance of P, and v2 is the variance of the distribution m* in M that is closest to P in KL divergence. This shows that prequential codes behave quite differently from other important universal codes such as the 2-part MDL, Shtarkov and Bayes codes, for which c=1. This behavior is undesirable in an MDL model selection setting.

Citations

• Present Position and Potential Developments: Some Personal Views: Statistical Theory: The Prequential Approach

Cited by

• Fine-Tuned In-Context Learners for Efficient Adaptation

Related

• Prequential Plug-In Codes that Achieve Optimal Redundancy Rates even if the Model is Wrong
• Improved MDL Estimators Using Fiber Bundle of Local Exponential Families for Non-exponential Families
• Upper Bound on Normalized Maximum Likelihood Codes for Gaussian Mixture Models
• Approximation of the Two-Part MDL Code
• Finite-sample risk bounds for maximum likelihood estimation with arbitrary penalties