Teaching Precommitted Agents: Model-Free Policy Evaluation and Control in Quasi-Hyperbolic Discounted MDPs

2025/09/07 by S. R. Eshwar, Eshwar, S. R.
Computer Science · Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Optimization and Search Problems #Reinforcement Learning in Robotics #Smart Grid Energy Management

paper · pdf · doi:10.48550/arxiv.2509.06094

openalex publication_date 2025/09/07 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

Time-inconsistent preferences, where agents favor smaller-sooner over larger-later rewards, are a key feature of human and animal decision-making. Quasi-Hyperbolic (QH) discounting provides a simple yet powerful model for this behavior, but its integration into the reinforcement learning (RL) framework has been limited. This paper addresses key theoretical and algorithmic gaps for precommitted agents with QH preferences. We make two primary contributions: (i) we formally characterize the structure of the optimal policy, proving for the first time that it reduces to a simple one-step non-stationary form; and (ii) we design the first practical, model-free algorithms for both policy evaluation and Q-learning in this setting, both with provable convergence guarantees. Our results provide foundational insights for incorporating QH preferences in RL.

Citations

• Reinforcement Learning with Quasi-Hyperbolic Discounting
• Stochastic Approximation: A Dynamical Systems Viewpoint
• Reinforcement Learning: An Introduction
• Myopia and Inconsistency in Dynamic Utility Maximization

Related

• Reinforcement Learning with Quasi-Hyperbolic Discounting
• Hyperbolic Discounting and Learning over Multiple Horizons
• Reinforcement Learning with Non-Exponential Discounting
• Rethinking the Discount Factor in Reinforcement Learning: A Decision\n Theoretic Approach
• Model-Free Non-Stationary RL: Near-Optimal Regret and Applications in Multi-Agent RL and Inventory Control