2025/01/15 by Yiran Tao, Jehan Yang, Tao, Yiran +5 · 2 citations
Computer Science · Engineering · Neuroscience · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Gaze Tracking and Assistive Technology #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Robotics (cs.RO) #Tactile and Sensory Interactions #Teleoperation and Haptic Systems
paper · pdf · doi:10.48550/arxiv.2501.08558
openalex publication_date 2025/01/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Teleoperating high degrees-of-freedom (DoF) robotic manipulators via low-DoF controllers like joysticks often requires frequent switching between control modes, where each mode maps controller movements to specific robot actions. Manually performing this frequent switching can make teleoperation cumbersome and inefficient. On the other hand, existing automatic mode-switching solutions, such as heuristic-based or learning-based methods, are often task-specific and lack generalizability. In this paper, we introduce LLM-Driven Automatic Mode Switching (LAMS), a novel approach that leverages Large Language Models (LLMs) to automatically switch control modes based on task context. Unlike existing methods, LAMS requires no prior task demonstrations and incrementally improves by integrating user-generated mode-switching examples. We validate LAMS through an ablation study and a user study with 10 participants on complex, long-horizon tasks, demonstrating that LAMS effectively reduces manual mode switches, is preferred over alternative methods, and improves performance over time. The project website with supplementary materials is at https://lams-assistance.github.io/.