Exploring the Approximation Capabilities of Multiplicative Neural Networks for Smooth Functions

2023/01/11 by Ido Ben-Shaul, Tomer Galanti, Ben-Shaul, Ido +3 · 1 citation
Computer Science · Physics and Astronomy · #41A25 #68Q32 #68T07 #Computational Physics and Python Applications #FOS: Computer and information sciences #FOS: Mathematics #Functional Analysis (math.FA) #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE)

paper · pdf · doi:10.48550/arxiv.2301.04605

openalex publication_date 2023/01/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Multiplication layers are a key component in various influential neural network modules, including self-attention and hypernetwork layers. In this paper, we investigate the approximation capabilities of deep neural networks with intermediate neurons connected by simple multiplication operations. We consider two classes of target functions: generalized bandlimited functions, which are frequently used to model real-world signals with finite bandwidth, and Sobolev-Type balls, which are embedded in the Sobolev Space W^r,2. Our results demonstrate that multiplicative neural networks can approximate these functions with significantly fewer layers and neurons compared to standard ReLU neural networks, with respect to both input dimension and approximation error. These findings suggest that multiplicative gates can outperform standard feed-forward layers and have potential for improving neural network design.

Cited by

• A New Perspective To Understanding Multi-resolution Hash Encoding For Neural Fields

Related

• Why Deep Neural Networks for Function Approximation?
• On the Complexity of Computing and Learning with Multiplicative Neural Networks
• Deep Neural Network Approximation Theory
• Nonlinear Approximation and (Deep) ReLU Networks
• Optimal Approximation Complexity of High-Dimensional Functions with Neural Networks