On the Effectiveness of Offline RL for Dialogue Response Generation

2023/07/23 by Paloma Sodhi, Felix Wu, Sodhi, Paloma +7 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2307.12425

openalex publication_date 2023/07/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A common training technique for language models is teacher forcing (TF). TF attempts to match human language exactly, even though identical meanings can be expressed in different ways. This motivates use of sequence-level objectives for dialogue response generation. In this paper, we study the efficacy of various offline reinforcement learning (RL) methods to maximize such objectives. We present a comprehensive evaluation across multiple datasets, models, and metrics. Offline RL shows a clear performance improvement over teacher forcing while not inducing training instability or sacrificing practical training budgets.

Cited by

• Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space

Related

• CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning
• Human-centric Dialog Training via Offline Reinforcement Learning
• Dialogue Evaluation with Offline Reinforcement Learning
• Improving Multi-Domain Task-Oriented Dialogue System with Offline Reinforcement Learning
• Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond