Ridge Regression: Biased Estimation for Nonorthogonal Problems

1970/02/01 by Arthur E. Hoerl, Robert W. Kennard · 9,132 citations
Engineering · Mathematics · #Advanced Statistical Methods and Models #Algorithm #Applied mathematics #Control Systems and Identification #Diagonal #Geology #Geometry #Least-squares function approximation #Mathematics #Regression #Residual #Ridge #Statistical and numerical algorithms #Statistics

paper · doi:10.1080/00401706.1970.10488634

published in Technometrics 12(1), 55-67 (Taylor & Francis)

openalex publication_date 1970/02/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

In multiple regression it is shown that parameter estimates based on minimum residual sum of squares have a high probability of being unsatisfactory, if not incorrect, if the prediction vectors are not orthogonal. Proposed is an estimation procedure based on adding small positive quantities to the diagonal of X′X. Introduced is the ridge trace, a method for showing in two dimensions the effects of nonorthogonality. It is then shown how to augment X′X to obtain biased estimates with smaller mean square error.

Cited by

• Generalized Network Psychometrics: Combining Network and Latent Variable Models
• GREEN: A lightweight architecture using learnable wavelets and Riemannian geometry for biomarker exploration with EEG signals
• Regularized estimation of large covariance matrices
• The composite absolute penalties family for grouped and hierarchical variable selection
• Generalized Inverses, Ridge Regression, Biased Linear Estimation, and Nonlinear Estimation
• Asymptotic properties of bridge estimators in sparse high-dimensional regression models
• High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking
• Firth‐Type Penalized Methods of the Modified Poisson and Least‐Squares Regression Analyses for Binary Outcomes
• Regularization, early-stopping and dreaming: A Hopfield-like setup to address generalization and overfitting
• Predicting key soil properties from Vis-NIR spectra by applying dual-wavelength indices transformations and stacking machine learning approaches
• South Arabia’s prehistoric monument landscape shows social resilience to climate change
• Stimulus-Driven Leakage in Naturalistic Neuroimaging
• Multiple Learning for Regression in big data
• There are a million ways to be a woman and a million ways to be a man: Gender differences across personality nuances and nations
• Collinearity: a review of methods to deal with it and a simulation study evaluating their performance
• Flowers occupy color‐space extremes: an anthocyanin‐derived theoretical floral color‐space approach
• Shrinkage priors for Bayesian penalized regression
• Cloud Regime Contributions to Recent Changes in Outgoing Radiation: A 22‐year Satellite Data Analysis
• Economic Predictions With Big Data: The Illusion of Sparsity
• Socio-economic impacts of different categories of state expenditure in urban and rural regions
• An inconvenient dataset: bias and inappropriate inference with the multilevel model
• Elastic Net Regularization Paths for All Generalized Linear Models
• Successful Explanations Start with Accurate Descriptions: Questionnaire Items as Personality Markers for More Accurate Predictions
• A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence
• Sample size for binary logistic prediction models: Beyond events per variable criteria
• Emulating human expertise in rezoning: a case study of the Australian statistical geography standard
• Development and validation of clinico‐imaging machine learning and deep learning models to predict responses to initial antiseizure medications in epilepsy
• Epistasis regulates genetic control of cardiac hypertrophy
• One-at-a-time knockoffs
• Machine Learning-Enabled Image Analysis of Complex Chemical Mixtures: Synthetic Urine Droplets as a Test System
• Tracking SARS-CoV-2 genomic variants in wastewater sequencing data with LolliPop
• Integrating genomic prediction and genotype specific parameter estimation in ecophysiological models: overview and perspectives
• Accelerating materials discovery for water crisis: Multi-objective machine learning for atmospheric water harvesting by MOFs
• Penalized regression with correlation-based penalty
• Accurate XANES determination of microscale Fe redox state in clinopyroxene: A multivariate approach with polarization-dependent Fe K -edge XAFS
• Probabilistic Forecasting in Day-Ahead Electricity Markets: Simulating Peak and Off-Peak Prices
• Fitness Effect of the Isoniazid Resistance Mutation S315T of the Catalase-Peroxidase Enzyme KatG of Mycobacterium tuberculosis
• Genome-wide association analysis of flowering date in a collection of cultivated olive tree
• Brain-inspired wiring economics for artificial neural networks
• A mathematical theory of relational generalization in transitive inference
• Super-resolution techniques to simulate electronic spectra of large molecular systems
• COVID-19 Future Forecasting Using Supervised Machine Learning Models
• A tutorial on individualized treatment effect prediction from randomized trials with a binary endpoint
• Why Should We Scan for Heterogeneous Treatment Effects?
• Making text count: Economic forecasting using newspaper text
• Semi-parametric genomic-enabled prediction of genetic values using reproducing kernel Hilbert spaces methods
• Penalized Regressions: The Bridge versus the Lasso
• Versatile Descent Algorithms for Group Regularization and Variable Selection in Generalized Linear Models
• microRNA‐based predictor for diagnosis of frontotemporal dementia
• Understanding Graduate School Through AI: A Scalable Approach to Thematic Coding
• Linear and nonlinear multidimensional functional connectivity methods reveal similar networks for semantic processing in EEG/MEG data
• Predicting persistent pain after total knee arthroplasty using different machine learning algorithms
• Spatial heterogeneity in machine learning-based poverty mapping: Where do models underperform?
• Artificial intelligence in the discovery and design of molecular semiconductors: a systematic review
• Behavioral profile predicts ethanol preference in adolescent mice, but not in adults: A machine learning approach
• Prediction of trumpet performance descriptors using machine learning
• Bayesian Regression Markets
• Machine Learning Advances for Time Series Forecasting
• Lasso Meets Horseshoe : A Survey
• A Survey of the S-Lemma
• Can fMRI functional connectivity index dynamic neural communication and cognition?
• Performance of some ridge estimators for the gamma regression model
• Liu estimation in generalized linear models: application on gamma distributed response variable
• Ridge-type shrinkage estimators in generalized linear models with an application to prostate cancer data
• Questioning the news about economic growth: Sparse forecasting using thousands of news-based sentiment values
• Predicting inflation with recurrent neural networks
• Machine learning for large-scale crop yield forecasting
• Relationship Between Body Mass Index and Fracture Risk Is Mediated by Bone Mineral Density
• Predicting policy funding allocation with Machine Learning
• In Silico High-Performance Liquid Chromatography Method Development via Machine Learning
• Truncated SVD methods for discrete linear ill-posed problems
• CLEAR: Covariant LEAst-square Re-fitting with applications to image restoration
• Tensor-on-tensor regression
• Asset Pricing and Machine Learning: A critical review
• ECONOMETRICS MEETS SENTIMENT: AN OVERVIEW OF METHODOLOGY AND APPLICATIONS
• Physicochemical modelling of the retention mechanism of temperature-responsive polymeric columns for HPLC through machine learning algorithms
• Predicting electronic screening for fast Koopmans spectral functional calculations
• Identifying Politically Connected Firms: A Machine Learning Approach*
• Spherical Slepian functions and the polar gap in geodesy
• Comparing two samples by penalized logistic regression
• Piecewise linear regularized solution paths
• Neural correlates of blood flow measured by ultrasound
• A Bayesian Algorithm for Reconstructing Climate Anomalies in Space and Time. Part II: Comparison with the Regularized Expectation–Maximization Algorithm
• Estimating a common period for a set of irregularly sampled functions with applications to periodic variable star data
• Kernel methods in machine learning
• Bayesian shrinkage methods for partially observed data with many predictors
• Peroxisomal lactate dehydrogenase is generated by translational readthrough in mammals
• Satisficing: Integrating Two Traditions
• The Geography of Recent Genetic Ancestry across Europe
• The Collinearity Problem in Linear Regression. The Partial Least Squares (PLS) Approach to Generalized Inverses
• Group least squares regression for linear models with strongly correlated predictor variables
• Partial least squares regression and projection on latent structure regression (PLS Regression)
• Direct and indirect least squares methods in continuous-time parameter estimation
• Graph Kernels: State-of-the-Art and Future Challenges
• Ridge regression coupled with a new uninformative variable elimination algorithm as a new descriptor screening method: Application of data reduction in QSAR study of some sulfonated derivatives as c-Met inhibitors
• Winter wheat yield prediction using convolutional neural networks from environmental and phenological data
• Structures and Assumptions: Strategies to Harness Gene × Gene and Gene × Environment Interactions in GWAS
• Selection of tuning parameters in bridge regression models via Bayesian information criterion
• Hybrid localized graph kernel for machine learning energy-related\n properties of molecules and solids
• Inferring Influence Networks from Longitudinal Bipartite Relational Data
• Approximate message passing for nonconvex sparse regularization with stability and asymptotic analysis
• Evaluating Multi‐Label Machine Learning Models for Smart Home Environments
• Predicting the global far-infrared SED of galaxies via machine learning techniques
• Ensemble Kalman inversion: a derivative-free technique for machine learning tasks
• Quantum processor-inspired machine learning in the biomedical sciences
• A general approach of least squares estimation and optimal filtering
• Forecasting asylum-related migration flows with machine learning and data at scale
• Predicting acute kidney injury at hospital re-entry using high-dimensional electronic health record data
• Automated design of heuristics for resource-constrained project scheduling problem via regression algorithms
• Heterogeneous Multilayer Generalized Operational Perceptron
• Exact model comparisons in the plausibility framework
• Efficient nonparametric inference on the effects of stochastic interventions under two‐phase sampling, with applications to vaccine efficacy trials
• Ridge regression and asymptotic minimax estimation over spheres of growing dimension
• Ridge regression [wikipedia]
• Prediction of non-alcoholic fatty-liver disease and liver fat content by serum molecular lipids. [europepmc]
• Evaluation of the lasso and the elastic net in genome-wide association studies. [europepmc]
• Peroxisomal lactate dehydrogenase is generated by translational readthrough in mammals. [europepmc]
• Characterizing a psychiatric symptom dimension related to deficits in goal-directed control. [europepmc]
• GSimp: A Gibbs sampler based left-censored missing value imputation approach for metabolomics studies. [europepmc]
• Associations of cumulative exposure to heavy metal mixtures with obesity and its comorbidities among U.S. adults in NHANES 2003-2014. [europepmc]
• Personalized evidence based medicine: predictive approaches to heterogeneous treatment effects. [europepmc]
• Challenges in the Integration of Omics and Non-Omics Data. [europepmc]
• Comparison and development of machine learning tools in the prediction of chronic kidney disease progression. [europepmc]
• Statistical predictions with glmnet. [europepmc]
• An overview of methods to address distinct research questions on environmental mixtures: an application to persistent organic pollutants and leukocyte telomere length. [europepmc]
• q2-sample-classifier: machine-learning tools for microbiome classification and regression. [europepmc]
• Brain age prediction using deep learning uncovers associated sequence variants. [europepmc]
• Microbiome Multi-Omics Network Analysis: Statistical Considerations, Limitations, and Opportunities. [europepmc]
• MICOM: Metagenome-Scale Modeling To Infer Metabolic Interactions in the Gut Microbiota. [europepmc]
• Stop codon context influences genome-wide stimulation of termination codon readthrough by aminoglycosides. [europepmc]
• Combining magnetoencephalography with magnetic resonance imaging enhances learning of surrogate-biomarkers. [europepmc]
• Metabolomics and Multi-Omics Integration: A Survey of Computational Methods and Resources. [europepmc]
• Short-term forecasting COVID-19 cumulative confirmed cases: Perspectives for Brazil. [europepmc]
• Machine Learning for Neural Decoding. [europepmc]
• Cerebrospinal fluid metabolomics identifies 19 brain-related phenotype associations. [europepmc]
• Confounding by Socioeconomic Status in Epidemiological Studies of Air Pollution and Health: Challenges and Opportunities. [europepmc]
• Improving the accuracy of single-trial fMRI response estimates using GLMsingle. [europepmc]
• AGILE platform: a deep learning powered approach to accelerate LNP development for mRNA delivery. [europepmc]
• Accurate predictions on small data with a tabular foundation model. [europepmc]

Related

• Ridge Regression: Biased Estimation for Nonorthogonal Problems
• Generalized Ridge Regression: Biased Estimation for Multiple Linear Regression Models
• Ridge Regression: Applications to Nonorthogonal Problems
• Generalized Ridge Regression: Applications to Nonorthogonal Linear Regression Models
• New General Ridge-Type Estimator in Linear Regression Models