DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation

2019/11/01 by Yizhe Zhang, Siqi Sun, Zhang, Yizhe +15 · 72 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.1911.00536

Accepted by ACL 2020 system demonstration

openalex publication_date 2019/11/01 · arxiv created 2020/05/02 · arxiv updated 2020/05/05 · openalex created_date 2020/07/02 · openalex updated_date 2026/07/28

Abstract

We present a large, tunable neural conversational response generation model, DialoGPT (dialogue generative pre-trained transformer). Trained on 147M conversation-like exchanges extracted from Reddit comment chains over a period spanning from 2005 through 2017, DialoGPT extends the Hugging Face PyTorch transformer to attain a performance close to human both in terms of automatic and human evaluation in single-turn dialogue settings. We show that conversational systems that leverage DialoGPT generate more relevant, contentful and context-consistent responses than strong baseline systems. The pre-trained model and training pipeline are publicly released to facilitate research into neural response generation and the development of more intelligent open-domain dialogue systems.

Citations

• Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization

Cited by

• KV Cache Recycling to Expand Usable Context Capacity in Low Parameter LLMs
• Kardia-R1: Unleashing LLMs to Reason toward Understanding and Empathy for Emotional Support via Rubric-as-Judge Reinforcement Learning
• Understanding and Mitigating Over-refusal for Large Language Models via Safety Representation
• Affective Multimodal Agents with Proactive Knowledge Grounding for Emotionally Aligned Marketing Dialogue
• A General Highly Accurate Online Planning Method Integrating Large Language Models into Nested Rollout Policy Adaptation for Dialogue Tasks
• Persona-Aware Alignment Framework for Personalized Dialogue Generation
• ECO Decoding: Entropy-Based Control for Controllability and Fluency in Controllable Dialogue Generation
• Fine-Tuning DialoGPT on Common Diseases in Rural Nepal for Medical Conversations
• COOPERA: Continual Open-Ended Human-Robot Assistance
• Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Setting
• Teacher Demonstrations in a BabyLM's Zone of Proximal Development for Contingent Multi-Turn Interaction
• Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation
• Limitations of Current Evaluation Practices for Conversational Recommender Systems and the Potential of User Simulation
• Less Diverse, Less Safe: The Indirect But Pervasive Risk of Test-Time Scaling in Large Language Models
• Following the TRACE: A Structured Path to Empathetic Response Generation with Multi-Agent Models
• Human vs. Agent in Task-Oriented Conversations
• LLM-Guided Co-Training for Text Classification
• A Framework for Generating Artificial Datasets to Validate Absolute and Relative Position Concepts
• AgentCTG: Harnessing Multi-Agent Collaboration for Fine-Grained Precise Control in Text Generation
• Automated Generation of Research Workflows from Academic Papers: A Full-text Mining Framework
• Vibe Check: Understanding the Effects of LLM-Based Conversational Agents' Personality and Alignment on User Perceptions in Goal-Oriented Tasks
• PersonaFuse: A Personality Activation-Driven Framework for Enhancing Human-LLM Interactions
• Dual Knowledge-Enhanced Two-Stage Reasoner for Multimodal Dialog Systems
• Chatbot To Help Patients Understand Their Health
• Few-Shot Query Intent Detection via Relation-Aware Prompt Learning
• AI-in-the-Loop: Privacy Preserving Real-Time Scam Detection and Conversational Scambaiting by Leveraging LLMs and Federated Learning
• MTalk-Bench: Evaluating Speech-to-Speech Models in Multi-Turn Dialogues via Arena-style and Rubrics Protocols
• Txt2Sce: Scenario Generation for Autonomous Driving System Testing Based on Textual Reports
• Do LLM Modules Generalize? A Study on Motion Generation for Autonomous Driving
• LobRA: Multi-tenant Fine-tuning over Heterogeneous Data
• REFRAG: Rethinking RAG based Decoding
• Continuously Steering LLMs Sensitivity to Contextual Knowledge with Proxy Models
• CARE: Contextual Adaptation of Recommenders for LLM-based Conversational Recommendation
• STEP: Stepwise Curriculum Learning for Context-Knowledge Fusion in Conversational Recommendation
• AdaptFlow: Adaptive Workflow Optimization via Meta-Learning
• Score Before You Speak: Improving Persona Consistency in Dialogue Generation using Response Quality Scores
• Improving Fine-Grained Emotion Detection in Text with BERT and GoEmotions
• Evaluating, Synthesizing, and Enhancing for Customer Support Conversation
• Can LLMs Generate High-Quality Task-Specific Conversations?
• Beyond Single Labels: Improving Conversational Recommendation through LLM-Powered Data Augmentation
• LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems
• DRIVE: Disfluency-Rich Synthetic Dialog Data Generation Framework for Intelligent Vehicle Environments
• Dr.Copilot: A Multi-Agent Prompt Optimized Assistant for Improving Patient-Doctor Communication in Romanian
• Chat-Ghosting: A Comparative Study of Methods for Auto-Completion in Dialog Systems
• LumiCRS: Asymmetric Contrastive Prototype Learning for Long-Tail Conversational Recommender Systems
• HyperSumm-RL: A Dialogue Summarization Framework for Modeling Leadership Perception in Social Robots
• Why Multi-Interest Fairness Matters: Hypergraph Contrastive Multi-Interest Learning for Fair Conversational Recommender System
• Towards the Holographic Characteristic of LLMs for Efficient Short-text Generation
• From Persona to Person: Enhancing the Naturalness with Multiple Discourse Relations Graph Learning in Personalized Dialogue Generation
• The Behavior Gap: Evaluating Zero-shot LLM Agents in Complex Task-Oriented Dialogs
• LoRA Users Beware: A Few Spurious Tokens Can Manipulate Your Finetuned Model
• Enhancing Medical Dialogue Generation through Knowledge Refinement and Dynamic Prompt Adjustment
• Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order
• Think Like a Person Before Responding: A Multi-Faceted Evaluation of Persona-Guided LLMs for Countering Hate
• V-VAE: A Variational Auto Encoding Framework Towards Fine-Grained Control over Human-Like Chat
• Validating the Effectiveness of a Large Language Model-based Approach for Identifying Children's Development across Various Free Play Settings in Kindergarten
• From Macro to Micro: Probing Dataset Diversity in Language Model Fine-Tuning
• MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators
• DoctorAgent-RL: A Multi-Agent Collaborative Reinforcement Learning System for Multi-Turn Clinical Dialogue
• MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems
• DialogXpert: Driving Intelligent and Emotion-Aware Conversations through Online Value-Based Reinforcement Learning with LLM Priors
• The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation
• FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Management
• Disambiguation in Conversational Question Answering in the Era of LLMs and Agents: A Survey
• Counterspeech the ultimate shield! Multi-Conditioned Counterspeech Generation through Attributed Prefix Learning
• Must Read: A Comprehensive Survey of Computational Persuasion
• On the Cost and Benefits of Training Context with Utterance or Full Conversation Training: A Comparative Stud
• Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding
• Are Pre-trained Language Models Knowledgeable to Ground Open Domain Dialogues?
• EEG-Based Brain-LLM Interface for Human Preference Aligned Generation
• TopicRefine: Joint Topic Prediction and Dialogue Response Generation for Multi-turn End-to-End Dialogue System
• An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation

Related

• Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators
• DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances
• PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
• Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterances
• End-to-end Conversation Modeling Track in DSTC6