2024/11/26 by Qiangqiang Mao, Yankai Cao, Mao, Qiangqiang +1 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Data Management and Algorithms #Data Visualization and Analytics #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Polynomial and algebraic computation
paper · pdf · doi:10.48550/arxiv.2411.17003
openalex publication_date 2024/11/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The prevailing mindset is that a single decision tree underperforms classic random forests in testing accuracy, despite its advantages in interpretability and lightweight structure. This study challenges such a mindset by significantly improving the testing accuracy of an oblique regression tree through our gradient-based entire tree optimization framework, making its performance comparable to the classic random forest. Our approach reformulates tree training as a differentiable unconstrained optimization task, employing a scaled sigmoid approximation strategy. To ameliorate numerical instability, we propose an algorithmic scheme that solves a sequence of increasingly accurate approximations. Additionally, a subtree polish strategy is implemented to reduce approximation errors accumulated across the tree. Extensive experiments on 16 datasets demonstrate that our optimized tree outperforms the classic random forest by an average of 2.03% improvements in testing accuracy.