Optimizing Performance of Feedforward and Convolutional Neural Networks through Dynamic Activation Functions

2023/08/10 by Chinmay Rane, Kanishka Tyagi, Rane, Chinmay +3
Computer Science · #Advanced Neural Network Applications #Anomaly Detection Techniques and Applications #Computational Engineering #FOS: Computer and information sciences #Finance #Machine Learning (cs.LG) #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #and Science (cs.CE)

paper · pdf · doi:10.48550/arxiv.2308.05724

openalex publication_date 2023/08/10 · openalex created_date 2023/08/12 · openalex updated_date 2026/07/28

Abstract

Deep learning training training algorithms are a huge success in recent years in many fields including speech, text,image video etc. Deeper and deeper layers are proposed with huge success with resnet structures having around 152 layers. Shallow convolution neural networks(CNN's) are still an active research, where some phenomena are still unexplained. Activation functions used in the network are of utmost importance, as they provide non linearity to the networks. Relu's are the most commonly used activation function.We show a complex piece-wise linear(PWL) activation in the hidden layer. We show that these PWL activations work much better than relu activations in our networks for convolution neural networks and multilayer perceptrons. Result comparison in PyTorch for shallow and deep CNNs are given to further strengthen our case.

Related

• Deep Global-Connected Net With The Generalized Multi-Piecewise ReLU Activation in Deep Learning
• Rethink ReLU to Training Better CNNs
• Deep Learning with S-shaped Rectified Linear Activation Units
• Activation Functions: Do They Represent A Trade-Off Between Modular Nature of Neural Networks And Task Performance
• Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units