Deep Learning with Eigenvalue Decay Regularizer

2016/04/24 by Oswaldo Ludwig, Ludwig, Oswaldo · 1 citation
Computer Science · #Neural Networks and Applications #Machine Learning and ELM #Face and Expression Recognition

paper · pdf · doi:10.48550/arxiv.1604.06985

Abstract

This paper extends our previous work on regularization of neural networks using Eigenvalue Decay by employing a soft approximation of the dominant eigenvalue in order to enable the calculation of its derivatives in relation to the synaptic weights, and therefore the application of back-propagation, which is a primary demand for deep learning. Moreover, we extend our previous theoretical analysis to deep neural networks and multiclass classification problems. Our method is implemented as an additional regularizer in Keras, a modular neural networks library written in Python, and evaluated in the benchmark data sets Reuters Newswire Topics Classification, IMDB database for binary sentiment classification, MNIST database of handwritten digits and CIFAR-10 data set for image classification.

Cited by

• Private Learning on Networks: Part II

Related

• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Deep Learning Meets Sparse Regularization: A Signal Processing Perspective
• Gradient Normalization & Depth Based Decay For Deep Learning
• NeuralEF: Deconstructing Kernels by Deep Neural Networks
• Heavy-Tailed Regularization of Weight Matrices in Deep Neural Networks