Multimodal Behavioral Patterns Analysis with Eye-Tracking and LLM-Based Reasoning

2025/07/24 by Dongyang Guo, Yasmeen Abdrabou, Guo, Dongyang +5
Computer Science · #Advanced Text Analysis Techniques #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2507.18252

openalex publication_date 2025/07/24 · openalex created_date 2025/10/16 · openalex updated_date 2026/07/28

Abstract

Eye-tracking data reveals valuable insights into users' cognitive states but is difficult to analyze due to its structured, non-linguistic nature. While large language models (LLMs) excel at reasoning over text, they struggle with temporal and numerical data. This paper presents a multimodal human-AI collaborative framework designed to enhance cognitive pattern extraction from eye-tracking signals. The framework includes: (1) a multi-stage pipeline using horizontal and vertical segmentation alongside LLM reasoning to uncover latent gaze patterns; (2) an Expert-Model Co-Scoring Module that integrates expert judgment with LLM output to generate trust scores for behavioral interpretations; and (3) a hybrid anomaly detection module combining LSTM-based temporal modeling with LLM-driven semantic analysis. Our results across several LLMs and prompt strategies show improvements in consistency, interpretability, and performance, with up to 50% accuracy in difficulty prediction tasks. This approach offers a scalable, interpretable solution for cognitive modeling and has broad potential in adaptive learning, human-computer interaction, and educational analytics.

Citations

• An LLM Agent for Automatic Geospatial Data Analysis
• Towards Explainable Traffic Flow Prediction with Large Language Models
• On Task and in Sync: Examining the Relationship between Gaze Synchrony and Self-Reported Attention During Video Lecture Learning
• Common 7B Language Models Already Possess Strong Math Capabilities
• Making Large Language Models Better Data Creators
• Explainability for Large Language Models: A Survey
• A Survey on Evaluation of Large Language Models
• A Survey on Evaluation of Large Language Models
• ChatGPT for good? On opportunities and challenges of large language models for education
• Large Language Models Can Self-Improve
• What Does it Mean for a Language Model to Preserve Privacy?
• Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunities
• Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
• Language Models are Few-Shot Learners
• A Coefficient of Agreement for Nominal Scales
• Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges

Related

• Multimodal LLM Augmented Reasoning for Interpretable Visual Perception Analysis
• CogSteer: Cognition-Inspired Selective Layer Intervention for Efficiently Steering Large Language Models
• Integrating Cognitive Processing Signals into Language Models: A Review of Advances, Applications and Future Directions
• GazeInterpreter: Parsing Eye Gaze to Generate Eye-Body-Coordinated Narrations
• Team ÚFAL at CMCL 2022 Shared Task: Figuring out the correct recipe for predicting Eye-Tracking features using Pretrained Language Models