Training Neural Networks with an algorithm for piecewise linear\n functions

2021/05/13 by Francisco Barahona, João Gonçalves, Barahona, Francisco +1
Computer Science · #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2105.06503

Abstract

We present experiments on training neural networks with an algorithm that was\noriginally designed as a subgradient method, namely the Volume Algorithm. We\ncompare with two of the most frequently used algorithms like Momentum and Adam.\nWe also compare with the newly proposed algorithm COCOB.\n

Citations

• A Stochastic Approximation Method
• Stochastic Estimation of the Maximum of a Regression Function
• An overview of gradient descent optimization algorithms

Related

• A Tutorial on Neural Networks and Gradient-free Training
• Learning Neural Networks with Two Nonlinear Layers in Polynomial Time
• The empirical size of trained neural networks
• Training Neural Networks at Any Scale
• Training Linear Neural Networks: Non-Local Convergence and Complexity Results