2020/09/29 by Justin H. Le, Andrew R. Teel, Le, Justin H. +1
Computer Science · Engineering · Mathematics · #Advanced Optimization Algorithms Research #FOS: Electrical engineering #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #Systems and Control (eess.SY) #electronic engineering #information engineering
paper · pdf · doi:10.48550/arxiv.2009.13770
openalex publication_date 2020/09/29 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
Momentum methods for convex optimization often rely on precise choices of\nalgorithmic parameters, based on knowledge of problem parameters, in order to\nachieve fast convergence, as well as to prevent oscillations that could\nseverely restrict applications of these algorithms to cyber-physical systems.\nTo address these issues, we propose two dynamical systems, named the Hybrid\nHeavy-Ball System and Hybrid-inspired Heavy-Ball System, which employ a\nfeedback mechanism for driving the momentum state toward zero whenever it\npoints in undesired directions. We describe the relationship between the\nproposed systems and their discrete-time counterparts, deriving conditions\nbased on linear matrix inequalities for ensuring exponential rates in both\ncontinuous time and discrete time. We provide numerical LMI results to\nillustrate the effects of our reset mechanisms on convergence rates in a\nsetting that simulates uncertainty of problem parameters. Finally, we\nnumerically demonstrate the efficiency and avoidance of oscillations of the\nproposed systems when solving both strongly convex and non-strongly convex\nproblems.\n