Designing Precise and Robust Dialogue Response Evaluators

2020/04/10 by Tianyu Zhao, Divesh Lala, Zhao, Tianyu +3 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2004.04908

openalex publication_date 2020/04/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Automatic dialogue response evaluator has been proposed as an alternative to automated metrics and human evaluation. However, existing automatic evaluators achieve only moderate correlation with human judgement and they are not robust. In this work, we propose to build a reference-free evaluator and exploit the power of semi-supervised training and pretrained (masked) language models. Experimental results demonstrate that the proposed evaluator achieves a strong correlation (> 0.6) with human judgement and generalizes robustly to diverse responses and corpora. We open-source the code and data in https://github.com/ZHAOTING/dialog-processing.

Citations

• Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median
• Sequence to Sequence Learning with Neural Networks
• DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
• The Curious Case of Neural Text Degeneration
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Cited by

• Angle-Optimized Partial Disentanglement for Multimodal Emotion Recognition in Conversation

Related

• Speaker Sensitive Response Evaluation Model
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
• Learning an Unreferenced Metric for Online Dialogue Evaluation
• Dialogue Evaluation with Offline Reinforcement Learning
• How To Evaluate Your Dialogue System: Probe Tasks as an Alternative for Token-level Evaluation Metrics