Adaptive random Fourier features with Metropolis sampling

2020/07/21 by Aku Kammonen, Jonas Kiessling, Kammonen, Aku +7 · 3 citations
Computer Science · Mathematics · #65C05 (Secondary) #65D15 (Primary) 65D40 #FOS: Mathematics #Numerical Analysis (math.NA) #cs.NA #math.NA #msc:65C05 #msc:65D15 #msc:65D40

paper · pdf · doi:10.48550/arxiv.2007.10683

arxiv created 2020/11/26 · arxiv updated 2020/11/30

Abstract

The supervised learning problem to determine a neural network approximation ℝ^d\ni x↦∑_k=1^Kβ_k e^{iω_k⋅ x} with one hidden layer is studied as a random Fourier features algorithm. The Fourier features, i.e., the frequencies ω_k∈ℝ^d, are sampled using an adaptive Metropolis sampler. The Metropolis test accepts proposal frequencies ω_k', having corresponding amplitudes β_k', with the probability min\1, (|β_k'|/|β_k|)^γ\, for a certain positive parameter γ, determined by minimizing the approximation error for given computational work. This adaptive, non-parametric stochastic method leads asymptotically, as K→∞, to equidistributed amplitudes |β_k|, analogous to deterministic adaptive algorithms for differential equations. The equidistributed amplitudes are shown to asymptotically correspond to the optimal density for independent samples in random Fourier features methods. Numerical evidence is provided in order to demonstrate the approximation properties and efficiency of the proposed algorithm. The algorithm is tested both on synthetic data and a real-world high-dimensional benchmark.

Cited by

• Convergence for adaptive resampling of random Fourier features
• An Adaptive Random Fourier Features approach Applied to Learning Stochastic Differential Equations
• Wind Field Reconstruction with Adaptive Random Fourier Features

Related

• Convergence for adaptive resampling of random Fourier features
• Adaptive Random Fourier Features Training Stabilized By Resampling With Applications in Image Regression
• Smaller generalization error derived for a deep residual neural network compared to shallow networks
• On the Error of Random Fourier Features
• No-Trick (Treat) Kernel Adaptive Filtering using Deterministic Features