2024/08/29 by Szeyi Chan, Chan, Szeyi, Shihan Fu +11 · 3 citations
Computer Science · #AI in Service Interactions #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Speech and dialogue systems
paper · pdf · doi:10.48550/arxiv.2408.16465
openalex publication_date 2024/08/29 · openalex created_date 2024/09/29 · openalex updated_date 2026/07/28
Recent progress in large language model (LLM) technology has significantly enhanced the interaction experience between humans and voice assistants (VAs). This project aims to explore a user's continuous interaction with LLM-based VA (LLM-VA) during a complex task. We recruited 12 participants to interact with an LLM-VA during a cooking task, selected for its complexity and the requirement for continuous interaction. We observed that users show both verbal and nonverbal behaviors, though they know that the LLM-VA can not capture those nonverbal signals. Despite the prevalence of nonverbal behavior in human-human communication, there is no established analytical methodology or framework for exploring it in human-VA interactions. After analyzing 3 hours and 39 minutes of video recordings, we developed an analytical framework with three dimensions: 1) behavior characteristics, including both verbal and nonverbal behaviors, 2) interaction stages--exploration, conflict, and integration--that illustrate the progression of user interactions, and 3) stage transition throughout the task. This analytical framework identifies key verbal and nonverbal behaviors that provide a foundation for future research and practical applications in optimizing human and LLM-VA interactions.