Improving Deep Neural Network with Multiple Parametric Exponential Linear Units

2016/06/01 by Yang Li, Chunxiao Fan, Li, Yang +7
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning and Data Classification #Machine Learning and ELM

paper · pdf · doi:10.48550/arxiv.1606.00305

openalex publication_date 2016/06/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Activation function is crucial to the recent successes of deep neural networks. In this paper, we first propose a new activation function, Multiple Parametric Exponential Linear Units (MPELU), aiming to generalize and unify the rectified and exponential linear units. As the generalized form, MPELU shares the advantages of Parametric Rectified Linear Unit (PReLU) and Exponential Linear Unit (ELU), leading to better classification performance and convergence property. In addition, weight initialization is very important to train very deep networks. The existing methods laid a solid foundation for networks using rectified linear units but not for exponential linear units. This paper complements the current theory and extends it to the wider range. Specifically, we put forward a way of initialization, enabling training of very deep networks using exponential linear units. Experiments demonstrate that the proposed initialization not only helps the training process but leads to better generalization performance. Finally, utilizing the proposed activation function and initialization, we present a deep MPELU residual architecture that achieves state-of-the-art performance on the CIFAR-10/100 datasets. The code is available at https://github.com/Coldmooon/Code-for-MPELU.

Citations

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• ImageNet Large Scale Visual Recognition Challenge
• A Fast Learning Algorithm for Deep Belief Nets
• ImageNet classification with deep convolutional neural networks
• Noisy Activation Functions
• Densely Connected Convolutional Networks
• Learning Multiple Layers of Features from Tiny Images

Related

• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Piecewise Linear Units Improve Deep Neural Networks
• Parametric Exponential Linear Unit for Deep Convolutional Neural Networks
• Overcoming Overfitting and Large Weight Update Problem in Linear Rectifiers: Thresholded Exponential Rectified Linear Units
• Natural-Logarithm-Rectified Activation Function in Convolutional Neural Networks