2014/05/13 by Manel Tagorti, Tagorti, Manel, Bruno Scherrer +1
Mathematics · Decision Sciences · Computer Science · #Markov Chains and Monte Carlo Methods #Advanced Bandit Algorithms Research #Reinforcement Learning in Robotics
paper · pdf · doi:10.48550/arxiv.1405.3229
We consider LSTD(λ), the least-squares temporal-difference algorithm with eligibility traces algorithm proposed by Boyan (2002). It computes a linear approximation of the value function of a fixed policy in a large Markov Decision Process. Under a β-mixing assumption, we derive, for any value of λ∈ (0,1), a high-probability estimate of the rate of convergence of this algorithm to its limit. We deduce a high-probability bound on the error of this algorithm, that extends (and slightly improves) that derived by Lazaric et al. (2012) in the specific case where λ=0. In particular, our analysis sheds some light on the choice of λ with respect to the quality of the chosen linear space and the number of samples, that complies with simulations.