A Formal Analysis of Multimodal Referring Strategies Under Common Ground

2020/03/16 by Nikhil Krishnaswamy, Krishnaswamy, Nikhil, James Pustejovsky +1 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multi-Agent Systems and Negotiation #Natural Language Processing Techniques #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2003.07385

openalex publication_date 2020/03/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we present an analysis of computationally generated mixed-modality definite referring expressions using combinations of gesture and linguistic descriptions. In doing so, we expose some striking formal semantic properties of the interactions between gesture and language, conditioned on the introduction of content into the common ground between the (computational) speaker and (human) viewer, and demonstrate how these formal features can contribute to training better models to predict viewer judgment of referring expressions, and potentially to the generation of more natural and informative referring expressions.

Cited by

• Dynamic Epistemic Friction in Dialogue
• Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetry

Related

• I see what you mean: Co-Speech Gestures for Reference Resolution in Multimodal Dialogue
• Collaborating on Referring Expressions
• Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions
• YouRefIt: Embodied Reference Understanding with Language and Gesture
• Multimodal encoding of motion events in speech, gesture and cognition