2017/08/06 by Thomas Kurbiel, Kurbiel, Thomas, Shahrzad Khaleghian +1 · 3 citations
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Face and Expression Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications #cs.AI #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1708.01911
6 pages, 14 figure
arxiv created 2017/08/06 · openalex publication_date 2017/08/06 · arxiv updated 2017/08/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The vanishing gradient problem was a major obstacle for the success of deep learning. In recent years it was gradually alleviated through multiple different techniques. However the problem was not really overcome in a fundamental way, since it is inherent to neural networks with activation functions based on dot products. In a series of papers, we are going to analyze alternative neural network structures which are not based on dot products. In this first paper, we revisit neural networks built up of layers based on distance measures and Gaussian activation functions. These kinds of networks were only sparsely used in the past since they are hard to train when using plain stochastic gradient descent methods. We show that by using Root Mean Square Propagation (RMSProp) it is possible to efficiently learn multi-layer neural networks. Furthermore we show that when appropriately initialized these kinds of neural networks suffer much less from the vanishing and exploding gradient problem than traditional neural networks even for deep networks.