2026/06/30 by Kushal Chakrabarti
#cs.LG #cs.AI #cs.CL
Bigger language models are less reliable. Across three families, three benchmarks and six rungs, including in-the-wild chat logs, scaling closes the start-of-response knowledge gap up to 7× while within-response knowledge degradation grows up to 39×. We trace that residual to one variable, the per-position disagreement δ= log pM - log pO against a stronger oracle, whose second moment splits exactly into bias2 KL(pM ‖ pO)2 and decoding risk Var[δ]. That split is an interpretability statement before it is a statistical one: the model's self-readable uncertainty H(pM) enters only the bias term, so the risk term has no model-readable component. Risk also takes a growing share of the squared error with scale, 31% to 49% from 1.7B to 14B. At a fabrication H(pM) relaxes within one token while risk persists up to 23× longer, leaving a confident-but-precarious regime that bridges consecutive fabrications (+69% at 14B). Contracting that risk at fixed KL removes 35-74% of web-verified hallucinations across six rungs and three families. Semantic entropy fires ≈30% less on that branch (p < 10-16) though it carries nearly 4× the fabrications. Bigger models snowball mistakes faster, through a failure mode that is dominant, self-perpetuating, causal and invisible to the model itself.