Nonconvex Nonsmooth Multicomposite Optimization and Its Applications to Recurrent Neural Networks

2025/06/22 by Jin, Lingzi, Wang, Xiao, Chen, Xiaojun
#FOS: Mathematics #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2506.17884

Abstract

We consider a class of nonconvex nonsmooth multicomposite optimization problems where the objective function consists of a Tikhonov regularizer and a composition of multiple nonconvex nonsmooth component functions. Such optimization problems arise from tangible applications in machine learning and beyond. To define and compute its first-order and second-order d(irectional)-stationary points effectively, we first derive the closed-form expression of the tangent cone for the feasible region of its constrained reformulation. Building on this, we establish its equivalence with the corresponding constrained and ℓ₁-penalty reformulations in terms of global optimality and d-stationarity. The equivalence offers indirect methods to attain the first-order and second-order d-stationary points of the original problem in certain cases. We apply our results to the training process of recurrent neural networks (RNNs).

Citations

• A four-operator splitting algorithm for nonconvex and nonsmooth optimization
• An Augmented Lagrangian Method for Training Recurrent Neural Networks
• Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees
• Proximal bundle methods for hybrid weakly convex composite optimization problems
• Gradient-Free Methods for Deterministic and Stochastic Nonsmooth Nonconvex Optimization
• An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsity
• Optimality conditions for nonsmooth nonconvex-nonconcave min-max problems and generative adversarial networks
• Linearly-constrained nonsmooth optimization for training autoencoders
• Complexity of Finding Stationary Points of Nonsmooth Nonconvex Functions
• Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learning
• Composite Difference-Max Programs for Modern Statistical Estimation Problems
• A Study of Piecewise Linear-Quadratic Programs
• Distributed optimization of deeply nested systems
• Long Short-Term Memory
• Finding Structure in Time

Related

• Stable Nonconvex-Nonconcave Training via Linear Interpolation
• Algorithms for solving optimization problems arising from deep neural net models: nonsmooth problems
• Efficient Methods for Structured Nonconvex-Nonconcave Min-Max\n Optimization
• Zeroth-Order Methods for Stochastic Nonconvex Nonsmooth Composite Optimization
• Complexity of Finding Stationary Points of Nonsmooth Nonconvex Functions