An Algorithm for Training Polynomial Networks

2013/04/26 by Roi Livni, Shai Shalev-Shwartz, Livni, Roi +3
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1304.7045

arxiv created 2014/02/20 · arxiv updated 2014/02/21

Abstract

We consider deep neural networks, in which the output of each node is a quadratic function of its inputs. Similar to other deep architectures, these networks can compactly represent any function on a finite training set. The main goal of this paper is the derivation of an efficient layer-by-layer algorithm for training such networks, which we denote as the Basis Learner. The algorithm is a universal learner in the sense that the training error is guaranteed to decrease at every iteration, and can eventually reach zero under mild conditions. We present practical implementations of this algorithm, as well as preliminary experimental results. We also compare our deep architecture to other shallow architectures for learning polynomials, in particular kernel learning.

Related

• A Polynomial-Based Approach for Architectural Design and Learning with\n Deep Neural Networks
• Ladder Polynomial Neural Networks
• SUPN: Shallow Universal Polynomial Networks
• Learning Polynomial Activation Functions for Deep Neural Networks
• Analysis of Deep Neural Networks with Quasi-optimal polynomial approximation rates