By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting

2024/07/15 by H.S. Yoon, Biniyam Aschalew Tolera, Yoon, Hyungjun +7 · 5 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2407.10385

openalex publication_date 2024/07/15 · openalex created_date 2024/07/17 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) have demonstrated exceptional abilities across various domains. However, utilizing LLMs for ubiquitous sensing applications remains challenging as existing text-prompt methods show significant performance degradation when handling long sensor data sequences. We propose a visual prompting approach for sensor data using multimodal LLMs (MLLMs). We design a visual prompt that directs MLLMs to utilize visualized sensor data alongside the target sensory task descriptions. Additionally, we introduce a visualization generator that automates the creation of optimal visualizations tailored to a given sensory task, eliminating the need for prior task-specific knowledge. We evaluated our approach on nine sensory tasks involving four sensing modalities, achieving an average of 10% higher accuracy than text-based prompts and reducing token costs by 15.8 times. Our findings highlight the effectiveness and cost-efficiency of visual prompts with MLLMs for various sensory tasks. The source code is available at https://github.com/diamond264/ByMyEyes.

Cited by

• LENS: LLM-Enabled Narrative Synthesis for Mental Health by Aligning Multimodal Sensing with Language Models
• Achieving Effective Virtual Reality Interactions via Acoustic Gesture Recognition based on Large Language Models
• SignalLLM: A General-Purpose LLM Agent Framework for Automated Signal Processing
• MobQA: A Benchmark Dataset for Semantic Understanding of Human Mobility Data through Question Answering
• ZARA: Zero-shot Motion Time-Series Analysis via Knowledge and Retrieval Driven LLM Agents

Related

• SensorBench: Benchmarking LLMs in Coding-Based Sensor Processing
• HoloLLM: Multisensory Foundation Model for Language-Grounded Human Sensing and Reasoning
• Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want
• SensorLM: Learning the Language of Wearable Sensors
• SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for Large-scale Vision-Language Models