Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks

2019/07/15 by Alejandro Molina, Patrick Schramowski, Molina, Alejandro +3 · 1 voice · 20 citations
Chemistry · Computer Science · Mathematics · #Activation function #Advanced Neural Network Applications #Artificial intelligence #Artificial neural network #Chemistry #Computer science #Deep learning #Domain Adaptation and Few-Shot Learning #End-to-end principle #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Mathematics #Neural and Evolutionary Computing (cs.NE) #Parametric statistics #Robustness (evolution) #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.1907.06732

published in arXiv (Cornell University) (Cornell University) · 17 Pages, 8 Figures

openalex publication_date 2019/07/15 · arxiv published 2019/07/15 · arxiv created 2020/02/04 · arxiv updated 2020/02/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

The performance of deep network learning strongly depends on the choice of the non-linear activation function associated with each neuron. However, deciding on the best activation is non-trivial, and the choice depends on the architecture, hyper-parameters, and even on the dataset. Typically these activations are fixed by hand before training. Here, we demonstrate how to eliminate the reliance on first picking fixed activation functions by using flexible parametric rational functions instead. The resulting Padé Activation Units (PAUs) can both approximate common activation functions and also learn new ones while providing compact representations. Our empirical evidence shows that end-to-end learning deep networks with PAUs can increase the predictive performance. Moreover, PAUs pave the way to approximations with provable robustness. https://github.com/ml-research/pau

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Gradient-based learning applied to document recognition
• ImageNet Large Scale Visual Recognition Challenge
• Learning Multiple Layers of Features from Tiny Images

Cited by

• DB-KAUNet: An Adaptive Dual Branch Kolmogorov-Arnold UNet for Retinal Vessel Segmentation
• When Swin Transformer Meets KANs: An Improved Transformer Architecture for Medical Image Segmentation
• Protenix-Mini+: efficient structure prediction model with scalable pairformer
• PO-CKAN:Physics Informed Deep Operator Kolmogorov Arnold Networks with Chunk Rational Structure
• Statistics of correlations in nonlinear recurrent neural networks
• Activation Function Design Sustains Plasticity in Continual Learning
• Jim137/qkan: v0.1.0
• Algebraic geometry of rational neural networks
• Neural network-based singularity detection and applications
• Human Body Weight Estimation Through Music-Induced Bed Vibrations
• AReLU: Attention-based Rectified Linear Unit
• Hi-fi functional priors by learning activations
• NLP Methods May Actually Be Better Than Professors at Estimating Question Difficulty
• Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning
• MNO : A Multi-modal Neural Operator for Parametric Nonlinear BVPs
• Padé Approximant Neural Networks for Enhanced Electric Motor Fault Diagnosis Using Vibration and Acoustic Data
• A parametric activation function based on Wendland RBF
• CoFrNets: Interpretable Neural Architecture Inspired by Continued Fractions
• FlashKAT: Understanding and Addressing Performance Bottlenecks in the Kolmogorov-Arnold Transformer
• KAN or MLP? Point Cloud Shows the Way Forward

Discussions

• Desafio qualquer um a ler esse artigo inteiro sem rir #5ªSérie arxiv.org/abs/1907.06732 [bsky, 0 points, 0 comments]

Related

• Orthogonal-Padé Activation Functions: Trainable Activation functions for smooth and faster convergence in deep networks
• DeepLABNet: End-to-end Learning of Deep Radial Basis Networks with Fully Learnable Basis Functions
• Activation function optimization method: Learnable series linear units (LSLUs)
• Deep Learning Activation Functions: Fixed-Shape, Parametric, Adaptive, Stochastic, Miscellaneous, Non-Standard, Ensemble
• Activation function design for deep networks: linearity and effective initialisation