2018/01/31 by Congzheng Song, Song, Congzheng, Yiming Sun +1
Computer Science · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1801.10273
arxiv created 2018/11/05 · arxiv updated 2018/11/06
Gaussian processes (GPs) are flexible models that can capture complex structure in large-scale dataset due to their non-parametric nature. However, the usage of GPs in real-world application is limited due to their high computational cost at inference time. In this paper, we introduce a new framework, kernel distillation, to approximate a fully trained teacher GP model with kernel matrix of size n× n for n training points. We combine inducing points method with sparse low-rank approximation in the distillation procedure. The distilled student GP model only costs O(m2) storage for m inducing points where m ≪ n and improves the inference time complexity. We demonstrate empirically that kernel distillation provides better trade-off between the prediction time and the test performance compared to the alternatives.