InfiniteBoost: building infinite ensembles with gradient descent

2017/06/04 by Alex Rogozhnikov, Tatiana Likhomanenko, Rogozhnikov, Alex +1
Computer Science · #FOS: Computer and information sciences #Image Retrieval and Classification Techniques #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Topological and Geometric Data Analysis

paper · pdf · doi:10.48550/arxiv.1706.01109

openalex publication_date 2017/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In machine learning ensemble methods have demonstrated high accuracy for the variety of problems in different areas. Two notable ensemble methods widely used in practice are gradient boosting and random forests. In this paper we present InfiniteBoost - a novel algorithm, which combines important properties of these two approaches. The algorithm constructs the ensemble of trees for which two properties hold: trees of the ensemble incorporate the mistakes done by others; at the same time the ensemble could contain the infinite number of trees without the over-fitting effect. The proposed algorithm is evaluated on the regression, classification, and ranking tasks using large scale, publicly available datasets.

Citations

• Stochastic gradient boosting

Related

• Explaining the Success of AdaBoost and Random Forests as Interpolating Classifiers
• Infinitesimal gradient boosting
• Infinite random forests for imbalanced classification tasks
• Exploiting random projections and sparsity with random forests and gradient boosting methods -- Application to multi-label and multi-output learning, random forest model compression and leveraging input sparsity
• Measuring the Algorithmic Convergence of Randomized Ensembles: The Regression Setting