2021/02/22 by Yangjun Ruan, Ruan, Yangjun, Karen Ullrich +14
Computer Science · Mathematics · #Advanced Data Compression Techniques #Artificial Intelligence (cs.AI) #Computation (stat.CO) #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #Information Theory (cs.IT) #Machine Learning (cs.LG) #cs.AI #cs.IT #cs.LG #math.IT #stat.CO
paper · pdf · doi:10.48550/arxiv.2102.11086
openalex publication_date 2021/02/22 · arxiv created 2021/06/15 · arxiv updated 2021/06/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Latent variable models have been successfully applied in lossless compression with the bits-back coding algorithm. However, bits-back suffers from an increase in the bitrate equal to the KL divergence between the approximate posterior and the true posterior. In this paper, we show how to remove this gap asymptotically by deriving bits-back coding algorithms from tighter variational bounds. The key idea is to exploit extended space representations of Monte Carlo estimators of the marginal likelihood. Naively applied, our schemes would require more initial bits than the standard bits-back coder, but we show how to drastically reduce this additional cost with couplings in the latent space. When parallel architectures can be exploited, our coders can achieve better rates than bits-back with little additional cost. We demonstrate improved lossless compression rates in a variety of settings, especially in out-of-distribution or sequential data compression.