q-Neurons: Neuron Activations based on Stochastic Jackson's Derivative Operators

2018/06/14 by Frank Nielsen, Ke Sun · 1 citation
Computer Science · Mathematics · Neuroscience · Physics and Astronomy · Psychology · #Activation function #Adversarial Robustness in Machine Learning #Artificial intelligence #Artificial neural network #Biology #Computer science #Derivative (finance) #Function (biology) #Model Reduction and Neural Networks #Neural Networks and Applications #Neuron #Neuroscience #Psychology #Scalability #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.1109/tnnls.2020.3005167

published in IEEE Transactions on Neural Networks and Learning Systems 32(6), 2782-2789 (Institute of Electrical and Electronics Engineers) · 12 pages, 5 figures, 1 table

arxiv created 2018/06/14 · openalex publication_date 2020/07/13 · arxiv updated 2021/04/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

We propose a new generic type of stochastic neurons, called q-neurons, that considers activation functions based on Jackson's q-derivatives with stochastic parameters q. Our generalization of neural network architectures with q-neurons is shown to be both scalable and very easy to implement. We demonstrate experimentally consistently improved performances over state-of-the-art standard activation functions, both on training and testing loss functions.

Citations

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Adding Gradient Noise Improves Learning for Very Deep Networks
• On the fundamental theorem of (p,q)-calculus and some (p,q)-Taylor formulas
• Enhanced q-Least Mean Square
• Gaussian Error Linear Units (GELUs)
• Sigmoid-weighted linear units for neural network function approximation in reinforcement learning
• Learning Multiple Layers of Features from Tiny Images

Cited by

• Q-derivative [wikipedia]

Related

• Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons
• q-RBFNN:A Quantum Calculus-based RBF Neural Network
• A New Type of Neurons for Machine Learning
• Learning Activation Functions to Improve Deep Neural Networks
• Learning Neural Activations