QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks

2025/09/28 by Qian Chen, Linxin Yang, Chen, Qian +7
Computer Science · #Artificial Intelligence (cs.AI) #Computational Physics and Python Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2510.03276

openalex publication_date 2025/09/28 · openalex created_date 2025/10/09 · openalex updated_date 2026/07/28

Abstract

The combination of linear transformations and non-linear activation functions forms the foundation of most modern deep neural networks, enabling them to approximate highly complex functions. This paper explores the introduction of quadratic transformations to further increase nonlinearity in neural networks, with the aim of enhancing the performance of existing architectures. To reduce parameter complexity and computational complexity, we propose a lightweight quadratic enhancer that uses low-rankness, weight sharing, and sparsification techniques. For a fixed architecture, the proposed approach introduces quadratic interactions between features at every layer, while only adding negligible amounts of additional model parameters and forward computations. We conduct a set of proof-of-concept experiments for the proposed method across three tasks: image classification, text classification, and fine-tuning large-language models. In all tasks, the proposed approach demonstrates clear and substantial performance gains.

Citations

• The Llama 3 Herd of Models
• QuadraNet V2: Efficient and Sustainable Training of High-Order Neural Networks with Quadratic Adaptation
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
• DoRA: Weight-Decomposed Low-Rank Adaptation
• QuadraNet: Improving High-Order Neural Interaction Efficiency with Hardware-Aware Quadratic Neural Networks
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
• One Neuron Saved Is One Neuron Earned: On Parametric Efficiency of Quadratic Networks
• LLaMA: Open and Efficient Foundation Language Models
• CVNets: High Performance Library for Computer Vision
• On Expressivity and Trainability of Quadratic Networks
• Transformers in Vision: A Survey
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects
• A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects
• GLU Variants Improve Transformer
• Graph Attention Networks
• Searching for Activation Functions
• Attention Is All You Need
• Pointer Sentinel Mixture Models
• Gaussian Error Linear Units (GELUs)
• Deep Residual Learning for Image Recognition
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• Sequence to Sequence Learning with Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• ImageNet classification with deep convolutional neural networks
• Long Short-Term Memory
• Backpropagation Applied to Handwritten Zip Code Recognition
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• QuadraNet: Improving High-Order Neural Interaction Efficiency with Hardware-Aware Quadratic Neural Networks
• Computational and Storage Efficient Quadratic Neurons for Deep Neural Networks
• QuadraNet V2: Efficient and Sustainable Training of High-Order Neural Networks with Quadratic Adaptation
• On Expressivity and Trainability of Quadratic Networks
• QuadraLib: A Performant Quadratic Neural Network Library for Architecture Optimization and Design Exploration