2026/03/15 by Deepon Halder, Raj Dabre · 1 voice
Computer Science · #Autoregressive model #Bandwidth (computing) #Decoding methods #Entropy (arrow of time) #Kullback–Leibler divergence #Natural Language Processing Techniques #Principle of maximum entropy #Probabilistic logic #Set (abstract data type) #Speech and dialogue systems #Stochastic process #Topic Modeling #cs.CL
paper · pdf · doi:10.48550/arxiv.2603.14567
openalex publication_date 2026/03/15 · arxiv published 2026/03/15 · arxiv updated 2026/03/15 · openalex created_date 2026/03/18 · openalex updated_date 2026/07/28
Probabilistic language generators are theoretically modeled as discrete stochastic processes, yet standard decoding strategies (Top-k, Top-p) impose static truncation rules that fail to accommodate the dynamic information density of natural language. This misalignment often forces a suboptimal trade-off: static bounds are either too restrictive for high-entropy creative generation or too permissive for low-entropy logical reasoning. In this work, we formalize the generation process as a trajectory through a relative probability manifold. We introduce Top-b (Adaptive Relative Band Sampling), a decoding strategy that regulates the candidate set via a dynamic bandwidth coefficient coupled strictly to the instantaneous Shannon entropy of the model's distribution. We provide a theoretical framework demonstrating that Top-b acts as a variance-minimizing operator on the tail distribution. Empirical validation on GPQA and GSM8K benchmarks indicates that Top-b significantly reduces generation entropy and inter-decoding variance while maintaining competitive reasoning accuracy, effectively approximating a self-regulating control system for autoregressive generation.