2020/06/04 by Vijay Pandey, Pandey, Vijay
Computer Science · Engineering · Mathematics · #Advanced Battery Technologies Research #Advanced Memory and Neural Computing #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #cs.CV #cs.LG #cs.NE #stat.ML
paper · pdf · doi:10.48550/arxiv.2006.02797
arxiv created 2020/06/04 · openalex publication_date 2020/06/04 · arxiv updated 2020/06/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In past few years, linear rectified unit activation functions have shown its significance in the neural networks, surpassing the performance of sigmoid activations. RELU (Nair & Hinton, 2010), ELU (Clevert et al., 2015), PRELU (He et al., 2015), LRELU (Maas et al., 2013), SRELU (Jin et al., 2016), ThresholdedRELU, all these linear rectified activation functions have its own significance over others in some aspect. Most of the time these activation functions suffer from bias shift problem due to non-zero output mean, and high weight update problem in deep complex networks due to unit gradient, which results in slower training, and high variance in model prediction respectively. In this paper, we propose, "Thresholded exponential rectified linear unit" (TERELU) activation function that works better in alleviating in overfitting: large weight update problem. Along with alleviating overfitting problem, this method also gives good amount of non-linearity as compared to other linear rectifiers. We will show better performance on the various datasets using neural networks, considering TERELU activation method compared to other activations.