TanhSoft -- a family of activation functions combining Tanh and Softplus

2020/09/08 by Koushik Biswas, Sandeep Kumar, Biswas, Koushik +5
Computer Science · #Advanced Neural Network Applications #Domain Adaptation and Few-Shot Learning #Adversarial Robustness in Machine Learning

paper · pdf · doi:10.48550/arxiv.2009.03863

Abstract

Deep learning at its core, contains functions that are composition of a linear transformation with a non-linear function known as activation function. In past few years, there is an increasing interest in construction of novel activation functions resulting in better learning. In this work, we propose a family of novel activation functions, namely TanhSoft, with four undetermined hyper-parameters of the form tanh(αx+βe^γx)ln(δ+e^x) and tune these hyper-parameters to obtain activation functions which are shown to outperform several well known activation functions. For instance, replacing ReLU with xtanh(0.6e^x)improves top-1 classification accuracy on CIFAR-10 by 0.46% for DenseNet-169 and 0.7% for Inception-v3 while with tanh(0.87x)ln(1 +e^x) top-1 classification accuracy on CIFAR-100 improves by 1.24% for DenseNet-169 and 2.57% for SimpleNet model.

Citations

• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Cited by

• Smooth Adversarial Training

Related

• Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark
• Parametric Leaky Tanh: A New Hybrid Activation Function for Deep Learning
• TaLU: A Hybrid Activation Function Combining Tanh and Rectified Linear Unit to Enhance Neural Networks
• ErfReLU: Adaptive Activation Function for Deep Neural Network
• K-TanH: Efficient TanH For Deep Learning