Approximation capability of neural networks on spaces of probability measures and tree-structured domains

2019/06/03 by Tomáš Pevný, Tomas Pevny, Vojtech Kovarik +3 · 3 citations
Computer Science · Mathematics · #FOS: Computer and information sciences #Fuzzy Logic and Control Systems #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Rough Sets and Fuzzy Logic #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1906.00764

openalex created_date 2019/03/22 · arxiv created 2019/06/03 · openalex publication_date 2019/06/03 · arxiv updated 2019/06/04 · openalex updated_date 2026/07/28

Abstract

This paper extends the proof of density of neural networks in the space of continuous (or even measurable) functions on Euclidean spaces to functions on compact sets of probability measures. By doing so the work parallels a more then a decade old results on mean-map embedding of probability measures in reproducing kernel Hilbert spaces. The work has wide practical consequences for multi-instance learning, where it theoretically justifies some recently proposed constructions. The result is then extended to Cartesian products, yielding universal approximation theorem for tree-structured domains, which naturally occur in data-exchange formats like JSON, XML, YAML, AVRO, and ProtoBuffer. This has important practical implications, as it enables to automatically create an architecture of neural networks for processing structured data (AutoML paradigms), as demonstrated by an accompanied library for JSON format.

Citations

• Multiple instance classification: Review, taxonomy and comparative study
• Solving the multiple instance problem with axis-parallel rectangles
• Learning from Distributions via Support Measure Machines
• Towards a Neural Statistician

Cited by

• Softmax as Linear Attention in the Large-Prompt Regime: a Measure-based Perspective
• Token Sample Complexity of Attention
• Adversarial Attack and Defense on Point Sets
• Measure-conditional Discriminator with Stationary Optimum for GANs and Statistical Distance Surrogates

Related

• Universal Approximation and the Topological Neural Network
• An Approximation Theory Perspective on Machine Learning
• Extending Universal Approximation Guarantees: A Theoretical Justification for the Continuity of Real-World Learning Tasks
• Distributionally robust approximation property of neural networks
• A Universal Approximation Theorem of Deep Neural Networks for Expressing Probability Distributions