Deep Maxout Network Gaussian Process

2022/08/08 by Libin Liang, Ye Tian, Liang, Libin +3
Computer Science · Engineering · Mathematics · #Control Systems and Identification #FOS: Computer and information sciences #G.3 #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.2208.04468

openalex publication_date 2022/08/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Study of neural networks with infinite width is important for better understanding of the neural network in practical application. In this work, we derive the equivalence of the deep, infinite-width maxout network and the Gaussian process (GP) and characterize the maxout kernel with a compositional structure. Moreover, we build up the connection between our deep maxout network kernel and deep neural network kernels. We also give an efficient numerical implementation of our kernel which can be adapted to any maxout rank. Numerical results show that doing Bayesian inference based on the deep maxout network kernel can lead to competitive results compared with their finite-width counterparts and deep neural network kernels. This enlightens us that the maxout activation may also be incorporated into other infinite-width neural network structures such as the convolutional neural network (CNN).

Related

• Deep Neural Networks as Gaussian Processes
• Wide Neural Networks with Bottlenecks are Deep Gaussian Processes
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• Neural Network Gaussian Processes by Increasing Depth
• Sparsity-depth Tradeoff in Infinitely Wide Deep Neural Networks