A Theory of Feature Learning in Kernel Models

2023/10/18 by Yunlu Chen, Yang Li, Chen, Yunlu +5
Computer Science · Mathematics · #FOS: Computer and information sciences #FOS: Mathematics #Face and Expression Recognition #Machine Learning (stat.ML) #Machine Learning and ELM #Optimization and Control (math.OC) #Statistical Methods and Inference #Statistics Theory (math.ST)

paper · pdf · doi:10.48550/arxiv.2310.11736

openalex publication_date 2023/10/18 · openalex created_date 2023/10/21 · openalex updated_date 2026/07/30

Abstract

We study feature learning in a compositional variant of kernel ridge regression in which the predictor is applied to a learnable linear transformation of the input. When the response depends on the input only through a low-dimensional predictive subspace, we show that all global minimizers of the population objective for the linear transformation annihilate directions orthogonal to this subspace, and in certain regimes, exactly identify the subspace. Moreover, we show that global minimizers of the finite-sample objective inherit the exact same low-dimensional structure with high probability, even without any explicit penalization on the linear transformation.

Related

• A Compositional Kernel Model for Feature Learning
• On the kernel learning problem
• Kernel Ridge Regression with Predicted Feature Inputs and Applications to Factor-Based Nonparametric Regression
• Interpretable Kernels
• A Geometrical Analysis of Kernel Ridge Regression and its Applications