Hyperbolic Discounting and Learning over Multiple Horizons

2019/02/19 by William Fedus, Fedus, William, Carles Gelada +7 · 1 voice · 3 citations
Computer Science · Mathematics · #Artificial Intelligence in Games #Computability, Logic, AI Algorithms #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mathematical and Theoretical Analysis #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1902.06865

openalex publication_date 2019/02/19 · arxiv published 2019/02/19 · arxiv created 2019/02/28 · arxiv updated 2019/03/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process. The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees of the Bellman equation. However, evidence from psychology, economics and neuroscience suggests that humans and animals instead have hyperbolic time-preferences. In this work we revisit the fundamentals of discounting in RL and bridge this disconnect by implementing an RL agent that acts via hyperbolic discounting. We demonstrate that a simple approach approximates hyperbolic discount functions while still using familiar temporal-difference learning techniques in RL. Additionally, and independent of hyperbolic discounting, we make a surprising discovery that simultaneously learning value functions over multiple time-horizons is an effective auxiliary task which often improves over a strong value-based RL agent, Rainbow.

Citations

• A framework for mesencephalic dopamine systems based on predictive Hebbian learning
• A Markovian Decision Process
• Planning and acting in partially observable stochastic domains
• A Distributional Perspective on Reinforcement Learning
• Many-Goals Reinforcement Learning
• Exploration by Random Network Distillation
• Rainbow: Combining Improvements in Deep Reinforcement Learning

Cited by

• The Formalism-Implementation Gap in Reinforcement Learning Research
• A Survey of State Representation Learning for Deep Reinforcement Learning
• Zero-Shot Reinforcement Learning Under Partial Observability
• Fast Efficient Hyperparameter Tuning for Policy Gradients

Discussions

• However, human and animals tend to show hyperbolic discounting. Far from being irrational, non-exponential discounting can be optimal depending on the uncertainty in the environment and arises in an a [bsky, 4 points, 1 comments]

Related

• Hyperbolically Discounted Temporal Difference Learning
• Reinforcement Learning with Non-Exponential Discounting
• Hyperbolically-Discounted Reinforcement Learning on Reward-Punishment Framework
• Teaching Precommitted Agents: Model-Free Policy Evaluation and Control in Quasi-Hyperbolic Discounted MDPs
• Reinforcement Learning with Quasi-Hyperbolic Discounting