2023/05/23 by Takyoung Kim, Kim, Takyoung, Jamin Shin +7 · 1 citation
Computer Science · Decision Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Personal Information Management and User Behavior #Speech and dialogue systems #Usability and User Interface Design
paper · pdf · doi:10.48550/arxiv.2305.13857
openalex publication_date 2023/05/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Most task-oriented dialogue (TOD) benchmarks assume users that know exactly how to use the system by constraining the user behaviors within the system's capabilities via strict user goals, namely "user familiarity" bias. This data bias deepens when it combines with data-driven TOD systems, as it is impossible to fathom the effect of it with existing static evaluations. Hence, we conduct an interactive user study to unveil how vulnerable TOD systems are against realistic scenarios. In particular, we compare users with 1) detailed goal instructions that conform to the system boundaries (closed-goal) and 2) vague goal instructions that are often unsupported but realistic (open-goal). Our study reveals that conversations in open-goal settings lead to catastrophic failures of the system, in which 92% of the dialogues had significant issues. Moreover, we conduct a thorough analysis to identify distinctive features between the two settings through error annotation. From this, we discover a novel "pretending" behavior, in which the system pretends to handle the user requests even though they are beyond the system's capabilities. We discuss its characteristics and toxicity while showing recent large language models can also suffer from this behavior.