Natural Language Reinforcement Learning

2024/02/11 by Xidong Feng, Feng, Xidong, Ziyu Wan +15 · 3 voices · 9 citations
Computer Science · Engineering · #Robotics and Automated Systems #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2402.07157

arxiv published 2024/02/11 · arxiv updated 2024/02/14

Abstract

Reinforcement Learning (RL) has shown remarkable abilities in learning policies for decision-making tasks. However, RL is often hindered by issues such as low sample efficiency, lack of interpretability, and sparse supervision signals. To tackle these limitations, we take inspiration from the human learning process and introduce Natural Language Reinforcement Learning (NLRL), which innovatively combines RL principles with natural language representation. Specifically, NLRL redefines RL concepts like task objectives, policy, value function, Bellman equation, and policy iteration in natural language space. We present how NLRL can be practically implemented with the latest advancements in large language models (LLMs) like GPT-4. Initial experiments over tabular MDPs demonstrate the effectiveness, efficiency, and also interpretability of the NLRL framework.

Cited by

• Reinforcement Learning via Self-Distillation
• Sophia: A Persistent Agent Framework of Artificial Life
• Cogito, Ergo Ludo: An Agent that Learns to Play by Reasoning and Planning
• Quantum Algorithms for Finite-horizon Markov Decision Processes
• LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacra
• Learning Stateful Predictive Knowledge From Experience
• SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning
• Learning from Language Feedback via Variational Policy Distillation
• Language-Critique Imitation Learning from Suboptimal Demonstrations

Discussions

• arxiv.org/abs/2402.07157 [bsky, 2 points, 0 comments]
• arxiv.org/abs/2402.07157 [bsky, 0 points, 0 comments]
• 今日読んだ論文 Natural Language Reinforcement Learning arxiv.org/abs/2402.07157 自然言語によるベルマンバックアップ（を主張するなにか）をやり始めた走りのような論文。実験は小規模なグリッド世界で、パラメータ的な学習もない状態で行われている。ここまで強化学習の枠組みに乗せようとしすぎるよりも、やはり自己蒸留的な方法での改善を目指したいよ [bsky, 0 points, 0 comments]

Related

• Natural Language Reinforcement Learning
• Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
• A Survey of Reinforcement Learning Informed by Natural Language
• Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs
• Natural Language-conditioned Reinforcement Learning with Inside-out Task Language Development and Translation