DRE: An Effective Dual-Refined Method for Integrating Small and Large Language Models in Open-Domain Dialogue Evaluation

2025/06/04 by Kun Zhao, Bohao Yang, Zhao, Kun +11 · 1 citation
Computer Science · Medicine · #Advanced Graph Neural Networks #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.04516

openalex publication_date 2025/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) excel at many tasks but struggle with ambiguous scenarios where multiple valid responses exist, often yielding unreliable results. Conversely, Small Language Models (SLMs) demonstrate robustness in such scenarios but are susceptible to misleading or adversarial inputs. We observed that LLMs handle negative examples effectively, while SLMs excel with positive examples. To leverage their complementary strengths, we introduce SLIDE (Small and Large Integrated for Dialogue Evaluation), a method integrating SLMs and LLMs via adaptive weighting. Building on SLIDE, we further propose a Dual-Refinement Evaluation (DRE) method to enhance SLM-LLM integration: (1) SLM-generated insights guide the LLM to produce initial evaluations; (2) SLM-derived adjustments refine the LLM's scores for improved accuracy. Experiments demonstrate that DRE outperforms existing methods, showing stronger alignment with human judgment across diverse benchmarks. This work illustrates how combining small and large models can yield more reliable evaluation tools, particularly for open-ended tasks such as dialogue evaluation.

Citations

• The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation
• Qwen Technical Report
• Improving Medical Dialogue Generation with Abstract Meaning Representations
• Enhancing Dialogue Generation via Dynamic Graph Knowledge Aggregation
• Large Language Models are not Fair Evaluators
• Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Information
• Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory
• Can Large Language Models Be an Alternative to Human Evaluations?
• Self-Refine: Iterative Refinement with Self-Feedback
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• GPT-4 Technical Report
• Is ChatGPT a Good NLG Evaluator? A Preliminary Study
• Large Language Models Are State-of-the-Art Evaluators of Translation Quality
• LLaMA: Open and Efficient Foundation Language Models
• GPTScore: Evaluate as You Desire
• Towards a Unified Multi-Dimensional Evaluator for Text Generation
• MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluation
• BARTScore: Evaluating Generated Text as Text Generation
• Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluation
• Improving Dialog Evaluation with a Multi-reference Adversarial Dataset\n and Large Scale Pretraining
• Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots
• Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• BERTScore: Evaluating Text Generation with BERT
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
• A Coefficient of Agreement for Nominal Scales

Cited by

• Why Text Prevails: Vision May Undermine Multimodal Medical Decision Making

Related

• SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation
• Emphasising Structured Information: Integrating Abstract Meaning Representation into LLMs for Enhanced Open-Domain Dialogue Evaluation
• Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluation
• Leveraging LLMs for Dialogue Quality Measurement
• LLM as a Scorer: The Impact of Output Order on Dialogue Evaluation