A duality framework for analyzing random feature and two-layer neural networks

2023/05/09 by Hongrui Chen, Jihao Long, Chen, Hongrui +3 · 1 citation
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2305.05642

openalex publication_date 2023/05/09 · openalex created_date 2023/05/12 · openalex updated_date 2026/07/28

Abstract

We consider the problem of learning functions within the F_p,π and Barron spaces, which play crucial roles in understanding random feature models (RFMs), two-layer neural networks, as well as kernel methods. Leveraging tools from information-based complexity (IBC), we establish a dual equivalence between approximation and estimation, and then apply it to study the learning of the preceding function spaces. The duality allows us to focus on the more tractable problem between approximation and estimation. To showcase the efficacy of our duality framework, we delve into two important but under-explored problems: 1) Random feature learning beyond kernel regime: We derive sharp bounds for learning F_p,π using RFMs. Notably, the learning is efficient without the curse of dimensionality for p>1. This underscores the extended applicability of RFMs beyond the traditional kernel regime, since F_p,π with p<2 is strictly larger than the corresponding reproducing kernel Hilbert space (RKHS) where p=2. 2) The L^∞ learning of RKHS: We establish sharp, spectrum-dependent characterizations for the convergence of L^∞ learning error in both noiseless and noisy settings. Surprisingly, we show that popular kernel ridge regression can achieve near-optimal performance in L^∞ learning, despite it primarily minimizing square loss. To establish the aforementioned duality, we introduce a type of IBC, termed I-complexity, to measure the size of a function class. Notably, I-complexity offers a tight characterization of learning in noiseless settings, yields lower bounds comparable to Le Cam's in noisy settings, and is versatile in deriving upper bounds. We believe that our duality framework holds potential for broad application in learning analysis across more scenarios.

Cited by

• Spectral connvergece of random feature method in one dimension

Related

• Minimum complexity interpolation in random features models
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks
• Multi-layer random features and the approximation power of neural networks
• Universal approximation property of Banach space-valued random feature models including random neural networks
• The Barron Space and the Flow-induced Function Spaces for Neural Network Models