vix.ing · top · new · best · stats

Greedy function approximation: A gradient boosting machine.

2001/10/01 by Jerome H. Friedman · 29,600 citations
Computer Science · Mathematics · Physics and Astronomy · #Applied mathematics #Artificial intelligence #Artificial neural network #Boosting (machine learning) #Computer science #Gradient boosting #Gradient descent #Logistic regression #Machine Learning and Algorithms #Mathematical optimization #Mathematics #Minification #Model Reduction and Neural Networks #Neural Networks and Applications #Random forest #Regression #Statistics

paper · pdf · doi:10.1214/aos/1013203451

published in The Annals of Statistics 29(5) (Institute of Mathematical Statistics)

openalex publication_date 2001/10/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Function estimation/approximation is viewed from the perspective of numerical optimization in function space, rather than parameter space. A connection is made between stagewise additive expansions and steepest-descent minimization. A general gradient descent “boosting” paradigm is developed for additive expansions based on any fitting criterion.Specific algorithms are presented for least-squares, least absolute deviation, and Huber-M loss functions for regression, and multiclass logistic likelihood for classification. Special enhancements are derived for the particular case where the individual additive components are regression trees, and tools for interpreting such “TreeBoost” models are presented. Gradient boosting of regression trees produces competitive, highly robust, interpretable procedures for both regression and classification, especially appropriate for mining less than clean data. Connections between this approach and the boosting methods of Freund and Shapire and Friedman, Hastie and Tibshirani are discussed.

Citations

Cited by