Neural networks and rational functions

2017/06/11 by Matus Telgarsky, Telgarsky, Matus · 22 citations
Computer Science · Mathematics · Physics and Astronomy · #Artificial intelligence #Artificial neural network #Combinatorics #Computer science #Degree (music) #Discrete mathematics #FOS: Computer and information sciences #Function (biology) #Fuzzy Logic and Control Systems #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mathematics #Model Reduction and Neural Networks #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Omega #Physics #Pure mathematics #Quantum mechanics #Rational design #Rational function #Representation (politics) #cs.LG #cs.NE #stat.ML

paper · pdf · doi:10.48550/arxiv.1706.03301

published in arXiv (Cornell University) (Cornell University) · To appear, ICML 2017

arxiv created 2017/06/11 · openalex publication_date 2017/06/11 · arxiv updated 2017/06/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Neural networks and rational functions efficiently approximate each other. In more detail, it is shown here that for any ReLU network, there exists a rational function of degree O(polylog(1/ε)) which is ε-close, and similarly for any rational function there exists a ReLU network of size O(polylog(1/ε)) which is ε-close. By contrast, polynomials need degree Ω(poly(1/ε)) to approximate even a single ReLU. When converting a ReLU network to a rational function as above, the hidden constants depend exponentially on the number of layers, which is shown to be tight; in other words, a compositional representation can be beneficial even for rational functions.

Citations

• Reliably Learning the ReLU in Polynomial Time
• Error bounds for approximations with deep ReLU networks

Cited by

• On the Design of Rational Polynomial State Feedback Controllers
• A Quantifier-Reversal Approximation Paradigm for Recurrent Neural Networks
• PO-CKAN:Physics Informed Deep Operator Kolmogorov Arnold Networks with Chunk Rational Structure
• Don't Forget the Nonlinearity: Unlocking Activation Functions in Efficient Fine-Tuning
• Neural network-based singularity detection and applications
• C¹-approximation with rational functions and rational neural networks
• On the algorithmic construction of deep ReLU networks
• Revealing Language Model Trajectories via Kullback-Leibler Divergence

Related

• Rational neural networks
• Representational Power of ReLU Networks and Polynomial Kernels: Beyond Worst-Case Analysis
• On the Expressiveness of Rational ReLU Neural Networks With Bounded Depth
• Nonlinear Approximation and (Deep) ReLU Networks
• C¹-approximation with rational functions and rational neural networks