2025/03/19 by Onno Eberhard, Michael Muehlebach, Eberhard, Onno +3 · 2 voices · 4 citations
Computer Science · #Reinforcement Learning in Robotics #cs.LG
paper · pdf · doi:10.48550/arxiv.2503.15200
openalex publication_date 2025/03/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Partially observable environments present a considerable computational challenge in reinforcement learning due to the need to consider long histories. Learning with a finite window of observations quickly becomes intractable as the window length grows. In this work, we introduce memory traces. Inspired by eligibility traces, these are compact representations of the history of observations in the form of exponential moving averages. We prove sample complexity bounds for the problem of offline on-policy evaluation that quantify the return errors achieved with memory traces for the class of Lipschitz continuous value estimates. We establish a close connection to the window approach, and demonstrate that, in certain environments, learning with memory traces is significantly more sample efficient. Finally, we underline the effectiveness of memory traces empirically in online reinforcement learning experiments for both value prediction and control.