Gradient directions and relative inexactness in optimization and machine learning

2024/06/30 by A.S. Vasin, Vasin, Artem · 1 citation
Engineering · #Advanced Measurement and Metrology Techniques #FOS: Mathematics #Optimization and Control (math.OC)

paper · pdf · doi:10.48550/arxiv.2407.00667

openalex publication_date 2024/06/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we investigate the influence of noise giving an estimate of the gradient having a acute angle with the original. Noise amplitude has a relative model. The work offers both theoretical calculations and theorems, as well as experimental results. Classic machine learning problems were chosen as experiments -- linear and logistic regression, computer vision and natural language processing.

Cited by

• Worst-case convergence analysis of relatively inexact gradient descent on smooth convex functions

Related

• Accelerated gradient methods with absolute and relative noise in the gradient
• Escaping Saddles with Stochastic Gradients
• Gradient descent in some simple settings
• Gradient descent in higher codimension
• The error accumulation in the conjugate gradient method for degenerate problem