A Risk Comparison of Ordinary Least Squares vs Ridge Regression

2011/05/04 by Dhillon, Paramveer S., Foster, Dean P., Kakade, Sham M. +1 · 1 citation
#FOS: Computer and information sciences #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1105.0875

Abstract

We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a Principal Component Analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that the risk of this ordinary least squares method is within a constant factor (namely 4) of the risk of ridge regression.

Cited by

• On the Interplay between Graph Structure and Learning Algorithms in Graph Neural Networks

Related

• Prediction Risk and Estimation Risk of the Ridgeless Least Squares Estimator under General Assumptions on Regression Errors
• Kernel ridge vs. principal component regression: minimax bounds and adaptability of regularization operators
• Risk Comparisons in Linear Regression: Implicit Regularization Dominates Explicit Regularization
• Ridge Regression: Applications to Nonorthogonal Problems
• Lecture notes on ridge regression