Fast training of large kernel models with delayed projections

2024/11/25 by Amirhesam Abedsoltan, Abedsoltan, Amirhesam, Siyuan Ma +5
Computer Science · #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Speech Recognition and Synthesis

paper · pdf · doi:10.48550/arxiv.2411.16658

openalex publication_date 2024/11/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Classical kernel machines have historically faced significant challenges in scaling to large datasets and model sizes--a key ingredient that has driven the success of neural networks. In this paper, we present a new methodology for building kernel machines that can scale efficiently with both data size and model size. Our algorithm introduces delayed projections to Preconditioned Stochastic Gradient Descent (PSGD) allowing the training of much larger models than was previously feasible, pushing the practical limits of kernel-based learning. We validate our algorithm, EigenPro4, across multiple datasets, demonstrating drastic training speed up over the existing methods while maintaining comparable or better classification accuracy.

Related

• Toward Large Kernel Models
• How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets
• Kernel machines that adapt to GPUs for effective large batch training
• Doubly stochastic large scale kernel learning with the empirical kernel map
• Majority Kernels: An Approach to Leverage Big Model Dynamics for Efficient Small Model Training