2020/08/17 by Philippe Goulet Coulombe, Coulombe, Philippe Goulet
Computer Science · #Anomaly Detection Techniques and Applications #Econometrics (econ.EM) #FOS: Computer and information sciences #FOS: Economics and business #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.2008.07063
openalex publication_date 2020/08/17 · openalex created_date 2022/10/01 · openalex updated_date 2026/07/28
It is notoriously difficult to build a bad Random Forest (RF). Concurrently, RF blatantly overfits in-sample without any apparent consequence out-of-sample. Standard arguments, like the classic bias-variance trade-off or double descent, cannot rationalize this paradox. I propose a new explanation: bootstrap aggregation and model perturbation as implemented by RF automatically prune a latent "true" tree. More generally, randomized ensembles of greedily optimized learners implicitly perform optimal early stopping out-of-sample. So there is no need to tune the stopping point. By construction, novel variants of Boosting and MARS are also eligible for automatic tuning. I empirically demonstrate the property, with simulated and real data, by reporting that these new completely overfitting ensembles perform similarly to their tuned counterparts -- or better.