2024/10/22 by Chenyi Li, Li, Chenyi, Guande Wu +13 · 10 citations
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #Context-Aware Activity Recognition Systems #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #IoT and Edge/Fog Computing
paper · pdf · doi:10.48550/arxiv.2410.16668
openalex publication_date 2024/10/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Augmented Reality (AR) assistance is increasingly used for supporting users with physical tasks like assembly and cooking. However, most systems rely on reactive responses triggered by user input, overlooking rich contextual and user-specific information. To address this, we present Satori, a novel AR system that proactively guides users by modeling both -- their mental states and environmental contexts. Satori integrates the Belief-Desire-Intention (BDI) framework with the state-of-the-art multi-modal large language model (LLM) to deliver contextually appropriate guidance. Our system is designed based on two formative studies involving twelve experts. We evaluated the system with a sixteen within-subject study and found that Satori matches the performance of designer-created Wizard-of-Oz (WoZ) systems, without manual configurations or heuristics, thereby improving generalizability, reusability, and expanding the potential of AR assistance.