DIAL: Direct Iterative Adversarial Learning for Realistic Multi-Turn Dialogue Simulation

2025/12/23 by Ziyi Zhu, Olivier Tieleman, Zhu, Ziyi +11
Computer Science · Psychology · #Computation and Language (cs.CL) #Digital Mental Health Interventions #FOS: Computer and information sciences #Speech and dialogue systems #Topic Modeling

paper · doi:10.48550/arxiv.2512.20773

openalex publication_date 2025/12/23 · openalex created_date 2025/12/26 · openalex updated_date 2026/07/28

Abstract

Realistic user simulation is crucial for training and evaluating multi-turn dialogue systems, yet creating simulators that accurately replicate human behavior remains a significant challenge. An effective simulator must expose the failure modes of the systems under evaluation. This work introduces Direct Iterative Adversarial Learning (DIAL), an adversarial framework that iteratively enhances user simulator realism through a competitive dynamic between a generator (user simulator) and a discriminator. When applied to mental health support, a domain characterized by diverse failure types and a critical dependence on realistic user behavior for failure detection, DIAL restores lexical diversity diminished by supervised fine-tuning and drastically reduces discriminator accuracy. The resulting simulator exhibits a strong correlation between simulated and real failure occurrence rates while maintaining low distributional divergence of failure modes. These findings indicate that DIAL is a promising method for developing realistic user simulators in multi-turn dialogue, facilitating reliable and cost-effective system evaluation prior to deployment.

Citations

• Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflection
• Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations
• The Llama 3 Herd of Models
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
• The Importance of Online Data: Understanding Preference Fine-tuning via Coverage
• Multi-turn Reinforcement Learning from Preference Human Feedback
• DuetSim: Building User Simulator with Dual Large Language Models for Task-Oriented Dialogues
• Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data
• ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL
• Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations
• Adversarial learning of neural user simulators for dialogue policy optimisation
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planning
• ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format
• Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator
• Controllable Dialogue Simulation with In-Context Learning
• GenTUS: Simulating User Behaviour and Language in Task-oriented Dialogues with Generative Transformers
• Offline RL for Natural Language Generation with Implicit Language Q Learning
• Training language models to follow instructions with human feedback
• Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems
• MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers
• Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward Decomposition
• Towards Scalable Multi-domain Conversational Agents: The Schema-Guided\n Dialogue Dataset
• MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for\n Task-Oriented Dialogue Modelling
• Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systems
• Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models
• Proximal Policy Optimization Algorithms
• Adversarial Learning for Neural Dialogue Generation
• A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systems
• Efficient RL for optimizing conversation level outcomes with an LLM-based tutor

Related

• MindEval: Benchmarking Language Models on Multi-turn Mental Health Support
• Adversarial learning of neural user simulators for dialogue policy optimisation
• Adversarial Training for High-Stakes Reliability
• Prompt Engineering a Schizophrenia Chatbot: Utilizing a Multi-Agent Approach for Enhanced Compliance with Prompt Instructions
• Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agents