Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient

2024/10/29 by Dinh, Vu C., Ho, Lam Si Tung, Nguyen, Cuong V. · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2410.22065

Abstract

We analyze the error rates of the Hamiltonian Monte Carlo algorithm with leapfrog integrator for Bayesian neural network inference. We show that due to the non-differentiability of activation functions in the ReLU family, leapfrog HMC for networks with these activation functions has a large local error rate of Ω(ε) rather than the classical error rate of O(ε³). This leads to a higher rejection rate of the proposals, making the method inefficient. We then verify our theoretical findings through empirical simulations as well as experiments on a real-world dataset that highlight the inefficiency of HMC inference on ReLU-based neural networks compared to analytical networks.

Cited by

• Smoothing Out Sticking Points: Sampling from Discrete-Continuous Mixtures with Dynamical Monte Carlo by Mapping Discrete Mass into a Latent Universe

Related

• Accelerating Hamiltonian Monte Carlo for Bayesian Inference in Neural Networks and Neural Operators
• Bayesian Inference with Latent Hamiltonian Neural Networks
• Variational Hamiltonian Monte Carlo via Score Matching
• HMC: avoiding rejections by not using leapfrog and some results on the acceptance rate
• Faster Hamiltonian Monte Carlo by Learning Leapfrog Scale