2025/12/01 by Yifeng Wu, Wu, Yudi, Wenhao Zhao +3
Computer Science · Neuroscience · Social Sciences · #Artificial Intelligence (cs.AI) #Embodied and Extended Cognition #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Language and cultural evolution #Machine Learning (cs.LG)
paper · pdf · doi:10.48550/arxiv.2512.01831
openalex publication_date 2025/12/01 · openalex created_date 2025/12/03 · openalex updated_date 2026/07/28
Generative diversity varies significantly across discrete latent generative models such as AR, MIM, and Diffusion. We propose a diagnostic framework, grounded in Information Bottleneck (IB) theory, to analyze the underlying strategies resolving this behavior. The framework models generation as a conflict between a 'Compression Pressure' - a drive to minimize overall codebook entropy - and a 'Diversity Pressure' - a drive to maximize conditional entropy given an input. We further decompose this diversity into two primary sources: 'Path Diversity', representing the choice of high-level generative strategies, and 'Execution Diversity', the randomness in executing a chosen strategy. To make this decomposition operational, we introduce three zero-shot, inference-time interventions that directly perturb the latent generative process and reveal how models allocate and express diversity. Application of this probe-based framework to representative AR, MIM, and Diffusion systems reveals three distinct strategies: "Diversity-Prioritized" (MIM), "Compression-Prioritized" (AR), and "Decoupled" (Diffusion). Our analysis provides a principled explanation for their behavioral differences and informs a novel inference-time diversity enhancement technique.