2021/07/24 by Balázs Kégl, Kégl, Balázs, Gabriel Hurtado +3 · 1 citation
Computer Science · Physics and Astronomy · #Artificial intelligence #Computer science #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Heteroscedasticity #Hyperparameter #Machine Learning (cs.LG) #Machine learning #Margin (machine learning) #Model Reduction and Neural Networks #Probabilistic logic #Reinforcement Learning in Robotics #Reinforcement learning #Schedule #Statistical model #cs.LG
paper · pdf · open access · doi:10.48550/arxiv.2107.11587
published in arXiv (Cornell University) (Cornell University) · Published at International Conference on Learning Representations, 2021: https://openreview.net/forum?id=p5uylG94S68
arxiv created 2021/07/24 · openalex publication_date 2021/07/24 · arxiv updated 2021/07/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We contribute to micro-data model-based reinforcement learning (MBRL) by rigorously comparing popular generative models using a fixed (random shooting) control agent. We find that on an environment that requires multimodal posterior predictives, mixture density nets outperform all other models by a large margin. When multimodality is not required, our surprising finding is that we do not need probabilistic posterior predictives: deterministic models are on par, in fact they consistently (although non-significantly) outperform their probabilistic counterparts. We also found that heteroscedasticity at training time, perhaps acting as a regularizer, improves predictions at longer horizons. At the methodological side, we design metrics and an experimental protocol which can be used to evaluate the various models, predicting their asymptotic performance when using them on the control problem. Using this framework, we improve the state-of-the-art sample complexity of MBRL on Acrobot by two to four folds, using an aggressive training schedule which is outside of the hyperparameter interval usually considered