Expressivity of Shallow Neural Networks Over Finite Fields

2026/07/19 by Maksym Zubkov, Carol Wu, Shiwei Yang +2
#math.AG #cs.NE

paper · pdf

Abstract

We study the expressivity of shallow polynomial neural networks (PNNs) with monomial activation functions over finite fields. For a given architecture, we define a neuromanifold as the image of the map from all possible network weights into the product of polynomial rings. We quantify the expressivity by the cardinality of the neuromanifold, and derive a natural lower and upper bound. This leads to counting rational points over finite fields, a problem closely linked to the Weil conjectures. Finally, we present an architecture that exhibits a striking difference in the neuromanifolds when considered over a characteristic zero versus a finite-characteristic field, illustrating the critical role of field characteristic in the notion of expressivity.

Citations

• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
• Orthogonal Matrices Over Finite Fields

Related

• On the Expressive Power of Deep Polynomial Neural Networks
• Activation degree thresholds and expressiveness of polynomial neural networks
• Algebraic geometry of rational neural networks
• Expressivity of Shallow and Deep Neural Networks for Polynomial Approximation
• The power of deeper networks for expressing natural functions