Reward Learning with Trees: Methods and Evaluation

2022/10/03 by Tom Bewley, Jonathan Lawry, Bewley, Tom +6
Computer Science · #Artificial Intelligence (cs.AI) #Bayesian Modeling and Causal Inference #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2210.01007

openalex publication_date 2022/10/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent efforts to learn reward functions from human feedback have tended to use deep neural networks, whose lack of transparency hampers our ability to explain agent behaviour or verify alignment. We explore the merits of learning intrinsically interpretable tree models instead. We develop a recently proposed method for learning reward trees from preference labels, and show it to be broadly competitive with neural networks on challenging high-dimensional tasks, with good robustness to limited or corrupted data. Having found that reward tree learning can be done effectively in complex settings, we then consider why it should be used, demonstrating that the interpretable reward structure gives significant scope for traceability, verification and explanation.

Related

• Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions
• Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback?
• Reward Generalization in RLHF: A Topological Perspective
• Reinforcement Learning from Diverse Human Preferences
• Understanding Learned Reward Functions