A New Training Method for Feedforward Neural Networks Based on Geometric Contraction Property of Activation Functions

2016/06/20 by Petre Birtea, Birtea, Petre, Cosmin Cernăzanu-Glăvan +3
Computer Science · Physics and Astronomy · #68T05 #92B20 #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Model Reduction and Neural Networks #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE)

paper · pdf · doi:10.48550/arxiv.1606.05990

openalex publication_date 2016/06/20 · openalex created_date 2016/07/22 · openalex updated_date 2026/07/28

Abstract

We propose a new training method for a feedforward neural network having the activation functions with the geometric contraction property. The method consists of constructing a new functional that is less nonlinear in comparison with the classical functional by removing the nonlinearity of the activation function from the output layer. We validate this new method by a series of experiments that show an improved learning speed and better classification error.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Improving neural networks by preventing co-adaptation of feature detectors
• Conditioning of quasi-Newton methods for function minimization
• A family of variable-metric methods derived by variational means
• Gradient-based learning applied to document recognition
• No More Pesky Learning Rates

Related

• Training Feedforward Neural Networks with Standard Logistic Activations\n is Feasible
• A Tutorial on Neural Networks and Gradient-free Training
• Data-Driven Learning of Feedforward Neural Networks with Different\n Activation Functions
• Geometric Decomposition of Feed Forward Neural Networks
• Lifted Neural Networks