OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts

2020/12/30 by Yuxian Meng, Shuhe Wang, Meng, Yuxian +11 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2012.15015

openalex publication_date 2020/12/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

When humans converse, what a speaker will say next significantly depends on what he sees. Unfortunately, existing dialogue models generate dialogue utterances only based on preceding textual contexts, and visual contexts are rarely considered. This is due to a lack of a large-scale multi-module dialogue dataset with utterances paired with visual contexts. In this paper, we release \bf OpenViDial, a large-scale multi-module dialogue dataset. The dialogue turns and visual contexts are extracted from movies and TV series, where each dialogue turn is paired with the corresponding visual context in which it takes place. OpenViDial contains a total number of 1.1 million dialogue turns, and thus 1.1 million visual contexts stored in images. Based on this dataset, we propose a family of encoder-decoder models leveraging both textual and visual contexts, from coarse-grained image features extracted from CNNs to fine-grained object features extracted from Faster R-CNNs. We observe that visual information significantly improves dialogue generation qualities, verifying the necessity of integrating multi-modal features for dialogue learning. Our work marks an important step towards large-scale multi-modal dialogue learning.

Citations

• Sequence to Sequence Learning with Neural Networks
• Data Noising as Smoothing in Neural Network Language Models
• Personalizing Dialogue Agents: I have a dog, do you have pets too?
• VQA: Visual Question Answering
• Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models
• Generating High-Quality and Informative Conversation Responses with Sequence-to-Sequence Models
• MUREL: Multimodal Relational Reasoning for Visual Question Answering
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• VD-BERT: A Unified Vision and Dialog Transformer with BERT

Cited by

• F2RVLM: Boosting Fine-grained Fragment Retrieval for Multi-Modal Long-form Dialogue with Vision Language Model

Related

• OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts
• ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialogue
• Constructing Multi-Modal Dialogue Dataset by Replacing Text with\n Semantically Relevant Images
• CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos
• InfoVisDial: An Informative Visual Dialogue Dataset by Bridging Large Multimodal and Language Models