Zero-Shot Reinforcement Learning via Function Encoders

2024/01/30 by Tyler Ingebrand, Amy Zhang, Ingebrand, Tyler +3 · 2 voices · 5 citations
#cs.LG #cs.AI

paper · pdf · doi:10.48550/arxiv.2401.17173

Abstract

Although reinforcement learning (RL) can solve many challenging sequential decision making problems, achieving zero-shot transfer across related tasks remains a challenge. The difficulty lies in finding a good representation for the current task so that the agent understands how it relates to previously seen tasks. To achieve zero-shot transfer, we introduce the function encoder, a representation learning algorithm which represents a function as a weighted combination of learned, non-linear basis functions. By using a function encoder to represent the reward function or the transition function, the agent has information on how the current task relates to previously seen tasks via a coherent vector representation. Thus, the agent is able to achieve transfer between related tasks at run time with no additional training. We demonstrate state-of-the-art data efficiency, asymptotic performance, and training stability in three RL fields by augmenting basic RL algorithms with a function encoder task representation.

Cited by

• Zero-Shot Function Encoder-Based Differentiable Predictive Control
• MoS-VLA: A Vision-Language-Action Model with One-Shot Skill Adaptation
• Towards Robust Zero-Shot Reinforcement Learning
• Function Spaces Without Kernels: Learning Compact Hilbert Space Representations
• Zero to Autonomy in Real-Time: Online Adaptation of Dynamics in Unstructured Environments

Discussions

• This is the simplest zero-shot RL baseline I could imagine and it seems to work really well. Really nice paper: arxiv.org/abs/2401.17173 [bsky, 63 points, 3 comments]
• Great read! arxiv.org/abs/2401.17173 [bsky, 1 points, 1 comments]

Related

• Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings
• A Unified Framework for Zero-Shot Reinforcement Learning
• Operator Deep Q-Learning: Zero-Shot Reward Transferring in Reinforcement Learning
• Towards Understanding the Benefit of Multitask Representation Learning in Decision Process
• Tackling the Zero-Shot Reinforcement Learning Loss Directly