2025/10/06 by Runlin Zhou, Zhou, Runlin, Chixiang Chen +3
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms
paper · pdf · doi:10.48550/arxiv.2510.05446
openalex publication_date 2025/10/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We study meta-reinforcement learning in finite-horizon MDPs where related tasks share similar structures in their optimal action-value functions. Specifically, we posit a linear representation Q^*h(s,a)=Φh(s,a) θ(k)h and place a Gaussian meta-prior N(θ^*h,Σ^*h) over the task-specific parameters θ(k)h. Building on randomized value functions, we propose two Thompson-style algorithms: (i) MTSRL, which learns only the prior mean and performs posterior sampling with the learned mean and known covariance; and (ii) MTSRL+, which additionally estimates the covariance and employs prior widening to control finite-sample estimation error. Further, we develop a prior-alignment technique that couples the posterior under the learned prior with a meta-oracle that knows the true prior, yielding meta-regret guarantees: we match prior-independent Thompson sampling in the small-task regime and strictly improve with more tasks once the prior is learned. Concretely, for known covariance we obtain O(H4S3/2√(ANK)) meta-regret, and with learned covariance O(H4S3/2√(AN3K)); both recover a better behavior than prior-independent after K \gtrsim O(H2) and K \gtrsim O(N2H2), respectively. Simulations on a stateful recommendation environment (with feature and prior misspecification) show that after brief exploration, MTSRL/MTSRL\(+\) track the meta-oracle and substantially outperform prior-independent RL and bandit-only meta-baselines. Our results give the first meta-regret guarantees for Thompson-style RL with learned Q-priors, and provide practical recipes (warm-start via RLSVI, OLS aggregation, covariance widening) for experiment-rich settings.