Norm-preserving Orthogonal Permutation Linear Unit Activation Functions\n (OPLU)

2016/04/08 by Artem Chernodub, Chernodub, Artem, Dimitri Nowicki +1 · 1 citation
Computer Science · Neuroscience · #Advanced Neural Network Applications #Brain Tumor Detection and Classification #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE)

paper · pdf · doi:10.48550/arxiv.1604.02313

openalex publication_date 2016/04/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We propose a novel activation function that implements piece-wise orthogonal\nnon-linear mappings based on permutations. It is straightforward to implement,\nand very computationally efficient, also it has little memory requirements. We\ntested it on two toy problems for feedforward and recurrent networks, it shows\nsimilar performance to tanh and ReLU. OPLU activation function ensures norm\npreservance of the backpropagated gradients, therefore it is potentially good\nfor the training of deep, extra deep, and recurrent neural networks.\n

Citations

• On the difficulty of training Recurrent Neural Networks
• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• Long Short-Term Memory
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• ImageNet classification with deep convolutional neural networks
• Unitary Evolution Recurrent Neural Networks
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Cited by

• Shifting Mean Activation Towards Zero with Bipolar Activation Functions

Related

• PLU: The Piecewise Linear Unit Activation Function
• Piecewise Linear Units Improve Deep Neural Networks
• Learning Activation Functions: A new paradigm for understanding Neural Networks
• Neural networks with trainable matrix activation functions
• Moderate Adaptive Linear Units (MoLU)