2026/07/26 by Oluwasegun A. Somefun
Computer Science · Engineering · #cs.LG #cs.AI #cs.SY #eess.SP #eess.SY
20 pages, 5 figures. Submitted to TMLR
arxiv created 2026/07/26 · arxiv updated 2026/08/06
In this paper, we develop a trust-region framework for understanding the behavior of adaptive moment estimation mechanisms, such as Adam, in stochastic gradient optimization. Specifically, in this framework, the magnitude of the update step for each individual weight is constrained within a trust-region governed by a moment constraint of order p∈[2,4]. The resulting derivation then leads to a family of learning-rate mechanisms based on second-moment estimation and a normalized p-th moment estimation. When p=4, this involves kurtosis-like estimation. The general mechanism, referred to as Gmake, provides a unified interpretation of normalization by moment estimation, learning-rate scheduling, spectral lowpass filtering as momentum, and operator-level spectral normalization within a common trust-region framework. Experiments on GPT2-124M trained on FineWeb-Edu and TinyStories suggest that the fourth-moment realization provides its greatest benefit when trust-region constraints are weak. As progressively stronger trust-region controls are introduced, the second-moment realization becomes increasingly competitive, often achieving slightly lower validation loss than its corresponding fourth-moment realization.