2021/10/11 by Rose E. Wang, Wang, Rose E., Julia White +5
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multiagent Systems (cs.MA) #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling #cs.AI #cs.CL #cs.LG #cs.MA
paper · pdf · doi:10.48550/arxiv.2110.05422
Findings of EMNLP 2021 Code: https://github.com/rosewang2008/calibrate_your_listeners
arxiv created 2021/10/11 · openalex publication_date 2021/10/11 · arxiv updated 2021/10/12 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
To be good conversational partners, natural language processing (NLP) systems should be trained to produce contextually useful utterances. Prior work has investigated training NLP systems with communication-based objectives, where a neural listener stands in as a communication partner. However, these systems commonly suffer from semantic drift where the learned language diverges radically from natural language. We propose a method that uses a population of neural listeners to regularize speaker training. We first show that language drift originates from the poor uncertainty calibration of a neural listener, which makes high-certainty predictions on novel sentences. We explore ensemble- and dropout-based populations of listeners and find that the former results in better uncertainty quantification. We evaluate both population-based objectives on reference games, and show that the ensemble method with better calibration enables the speaker to generate pragmatic utterances while scaling to a large vocabulary and generalizing to new games and listeners.