2020/07/29 by Sebastian Bruch, Bruch, Sebastian, Jan Pfeifer +4 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)
paper · pdf · doi:10.48550/arxiv.2007.14761
openalex publication_date 2020/07/29 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28
Axis-aligned decision forests have long been the leading class of machine\nlearning algorithms for modeling tabular data. In many applications of machine\nlearning such as learning-to-rank, decision forests deliver remarkable\nperformance. They also possess other coveted characteristics such as\ninterpretability. Despite their widespread use and rich history, decision\nforests to date fail to consume raw structured data such as text, or learn\neffective representations for them, a factor behind the success of deep neural\nnetworks in recent years. While there exist methods that construct smoothed\ndecision forests to achieve representation learning, the resulting models are\ndecision forests in name only: They are no longer axis-aligned, use stochastic\ndecisions, or are not interpretable. Furthermore, none of the existing methods\nare appropriate for problems that require a Transfer Learning treatment. In\nthis work, we present a novel but intuitive proposal to achieve representation\nlearning for decision forests without imposing new restrictions or\nnecessitating structural changes. Our model is simply a decision forest,\npossibly trained using any forest learning algorithm, atop a deep neural\nnetwork. By approximating the gradients of the decision forest through input\nperturbation, a purely analytical procedure, the decision forest directs the\nneural network to learn or fine-tune representations. Our framework has the\nadvantage that it is applicable to any arbitrary decision forest and that it\nallows the use of arbitrary deep neural networks for representation learning.\nWe demonstrate the feasibility and effectiveness of our proposal through\nexperiments on synthetic and benchmark classification datasets.\n