Unsupervised Meta-Testing with Conditional Neural Processes for Hybrid Meta-Reinforcement Learning

2025/06/04 by Ada, Suzan Ece, Ugur, Emre · 2 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics (cs.RO)

paper · doi:10.48550/arxiv.2506.04399

Abstract

We introduce Unsupervised Meta-Testing with Conditional Neural Processes (UMCNP), a novel hybrid few-shot meta-reinforcement learning (meta-RL) method that uniquely combines, yet distinctly separates, parameterized policy gradient-based (PPG) and task inference-based few-shot meta-RL. Tailored for settings where the reward signal is missing during meta-testing, our method increases sample efficiency without requiring additional samples in meta-training. UMCNP leverages the efficiency and scalability of Conditional Neural Processes (CNPs) to reduce the number of online interactions required in meta-testing. During meta-training, samples previously collected through PPG meta-RL are efficiently reused for learning task inference in an offline manner. UMCNP infers the latent representation of the transition dynamics model from a single test task rollout with unknown parameters. This approach allows us to generate rollouts for self-adaptation by interacting with the learned dynamics model. We demonstrate our method can adapt to an unseen test task using significantly fewer samples during meta-testing than the baselines in 2D-Point Agent and continuous control meta-RL benchmarks, namely, cartpole with unknown angle sensor bias, walker agent with randomized dynamics parameters.

Citations

• Autoregressive Conditional Neural Processes
• Contrastive Conditional Neural Processes
• A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning
• Generative Adversarial Networks with Conditional Neural Movement Primitives for An Interactive Generative Drawing Tool
• RMA: Rapid Motor Adaptation for Legged Robots
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning
• Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic\n Context Variables
• NoRML: No-Reward Meta Learning
• Attentive Neural Processes
• ProMP: Proximal Meta-Policy Search
• Conditional Neural Processes
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Proximal Policy Optimization Algorithms
• One-Shot Imitation Learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• OpenAI Gym
• High-Dimensional Continuous Control Using Generalized Advantage\n Estimation
• Playing Atari with Deep Reinforcement Learning
• Reinforcement Learning: An Introduction

Cited by

• Forecasting in Offline Reinforcement Learning for Non-stationary Environments
• A Survey of Continual Reinforcement Learning

Related

• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic\n Context Variables
• Meta-World Conditional Neural Processes
• On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning
• Meta-Reinforcement Learning by Tracking Task Non-stationarity
• Meta-Reinforcement Learning in Broad and Non-Parametric Environments