2026/07/16 by Hyemin Gu, Michael Tyrrell, Tuhin Sahai +1
#cs.LG #stat.ML
We propose the sublinear-growth principle for deep residual architectures -- a sharp stability threshold on the input-magnitude exponent of every residual block's velocity field: ‖v(x, t)‖ ≤ c ‖x‖q + b, q ∈ [0, 1]. The threshold q = 1 is established via two independent arguments. Classical ODE theory gives a global forward flow on [0, T] at q ≤ 1 and exhibits divergent velocity fields at any q > 1. The optimal-control analysis, via the Hamilton-Jacobi-Bellman equation, sharpens this to a selection statement: the training optimum is bang-bang on the boundary of the admissible class, so the optimum at q > 1 blows up while the optimum at q ≤ 1 is safe by construction. The exponent criterion q ≤ 1 is thereby a necessary and sufficient condition for stable training. It clarifies architectural placements that ensure the stability of training and inference, explaining, for instance, the stabilizing role of layer normalization. The sublinear-growth velocity fields form the right function space on which forward dynamics, adjoint sensitivity, and architectural composition are all well-controlled. An arithmetic of input-magnitude exponents under the five operations that build residual blocks enables efficient certification of qk ≤ 1 at the level of architectural primitives, in place of ad hoc trial and error in the search for stable neural architectural designs. A parameter-free modification reduces the supercritical Mamba block from q = 5 to q = 1 without layer normalization, demonstrating this point. Experiments on Mamba and PatchTST confirm that the q ≤ 1 variants train stably: the criterion is the input-magnitude exponent, not the presence of a normalization layer.