Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks

2017/02/20 by Chunjie Luo, Luo, Chunjie, Jianfeng Zhan +5 · 7 citations
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.1702.05870

openalex publication_date 2017/02/20 · openalex created_date 2019/06/27 · openalex updated_date 2026/07/28

Abstract

Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change of input distribution, thus results in poor generalization, and aggravates the internal covariate shift which slows down the training. To bound dot product and decrease the variance, we propose to use cosine similarity or centered cosine similarity (Pearson Correlation Coefficient) instead of dot product in neural networks, which we call cosine normalization. We compare cosine normalization with batch, weight and layer normalization in fully-connected neural networks as well as convolutional networks on the data sets of MNIST, 20NEWS GROUP, CIFAR-10/100 and SVHN. Experiments show that cosine normalization achieves better performance than other normalization techniques.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Gradient-based learning applied to document recognition
• A Fast Learning Algorithm for Deep Belief Nets
• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
• Deep Residual Learning for Image Recognition

Cited by

• Domain-constrained Synthesis of Inconsistent Key Aspects in Textual Vulnerability Descriptions
• Sharp Eyes and Memory for VideoLLMs: Information-Aware Visual Token Pruning for Efficient and Reliable VideoLLM Reasoning
• Connecting Giants: Synergistic Knowledge Transfer of Large Multimodal Models for Few-Shot Learning
• Lightweight and Generalizable Acoustic Scene Representations via Contrastive Fine-Tuning and Distillation
• Hyperspherical Prototype Networks
• Rethinking Layer-wise Model Merging through Chain of Merges
• von Mises-Fisher Mixture Model-based Deep learning: Application to Face Verification

Related

• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
• Weight Conditioning for Smooth Optimization of Neural Networks
• Training Deep Neural Networks Without Batch Normalization
• Filtered Batch Normalization
• Root Mean Square Layer Normalization