2019/06/13 by Sina Zarrieß, David Schlangen, Zarrieß, Sina +1 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling
paper · pdf · doi:10.48550/arxiv.1906.05518
openalex publication_date 2019/06/13 · openalex created_date 2022/07/28 · openalex updated_date 2026/07/28
Zero-shot learning in Language & Vision is the task of correctly labelling\n(or naming) objects of novel categories. Another strand of work in L&V aims at\npragmatically informative rather than ``correct'' object descriptions, e.g. in\nreference games. We combine these lines of research and model zero-shot\nreference games, where a speaker needs to successfully refer to a novel object\nin an image. Inspired by models of "rational speech acts", we extend a neural\ngenerator to become a pragmatic speaker reasoning about uncertain object\ncategories. As a result of this reasoning, the generator produces fewer nouns\nand names of distractor categories as compared to a literal speaker. We show\nthat this conversational strategy for dealing with novel objects often improves\ncommunicative success, in terms of resolution accuracy of an automatic\nlistener.\n