DIET: Lightweight Language Understanding for Dialogue Systems

2020/04/21 by Tanja Bunk, Bunk, Tanja, Daksh Varshneya +5 · 5 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2004.09936

openalex publication_date 2020/04/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large-scale pre-trained language models have shown impressive results on language understanding benchmarks like GLUE and SuperGLUE, improving considerably over other pre-training methods like distributed representations (GloVe) and purely supervised approaches. We introduce the Dual Intent and Entity Transformer (DIET) architecture, and study the effectiveness of different pre-trained representations on intent and entity prediction, two common dialogue language understanding tasks. DIET advances the state of the art on a complex multi-domain NLU dataset and achieves similarly high performance on other simpler datasets. Surprisingly, we show that there is no clear benefit to using large pre-trained models for this task, and in fact DIET improves upon the current state of the art even in a purely supervised setup without any pre-trained embeddings. Our best performing model outperforms fine-tuning BERT and is about six times faster to train.

Citations

• Partially observable Markov decision processes for spoken dialog systems
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• DocBERT: BERT for Document Classification
• PatentBERT: Patent Classification with Fine-Tuning a pre-trained BERT Model

Cited by

• Chatbots to strengthen democracy: An interdisciplinary seminar to train identifying argumentation techniques of science denial
• Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes
• A Sequence to Sequence Model for Extracting Multiple Product Name Entities from Dialog
• Do as I mean, not as I say: Sequence Loss Training for Spoken Language Understanding
• Multi-stage Clarification in Conversational AI: The case of\n Question-Answering Dialogue Systems

Related

• ConvFiT: Conversational Fine-Tuning of Pretrained Language Models
• ECLM: Entity Level Language Model for Spoken Language Understanding with Chain of Intent
• Spoken Language Understanding for Conversational AI: Recent Advances and Future Direction
• Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUE
• Joint model for intent and entity recognition