State of the Art of LLM-Enabled Interaction with Visualization
2026/01/21 by Mathis Brossier, Tobias Isenberg, Konrad Schönborn +5 · 1 voice
#cs.HC
paper · pdf
Abstract
We report on a systematic, PRISMA-guided survey of research at the intersection of LLMs and visualization, with a particular focus on visio-verbal interaction -- where verbal and visual modalities converge to support data sense-making. The emergence of Large Language Models (LLMs) has introduced new paradigms for interacting with data visualizations through natural language, leading to intuitive, multimodal, and accessible interfaces. We analyze 48 papers across six dimensions: application domain, visualization task, visualization representation, interaction modality, LLM integration, and system evaluation. Our classification framework maps LLM roles across the visualization pipeline, from data querying and transformation to visualization generation, explanation, and navigation. We highlight emerging design patterns, identify gaps in accessibility and visualization reading, and discuss the limitations of current LLMs in spatial reasoning and contextual grounding. We further reflect on evaluations of combined LLM-visualization systems, highlighting how current research projects tackle this challenge and discuss current gaps in conducting meaningful evaluations of such systems. With our survey we aim to guide future research and system design in LLM-enhanced visualization, supporting broad audiences and intelligent, conversational interfaces.
Citations
- Large Language Models for Software Engineering: A Reproducibility Crisis
- ConCISE: A Reference-Free Conciseness Evaluation Metric for LLM-Generated Answers
- Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies
- Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video
- The Era of Real-World Human Interaction: RL from User Conversations
- Your Model Is Unfair, Are You Even Aware? Inverse Relationship Between Comprehension and Trust in Explainability Visualizations of Biased ML Models
- Procedural Memory Is Not All You Need
- Evaluating LLMs for Visualization Generation and Understanding
- nvBench 2.0: Resolving Ambiguity in Text-to-Visualization through Stepwise Reasoning
- No LLM is Free From Bias: A Comprehensive Study of Bias Evaluation in Large Language Models
- NLI4DB: A Systematic Review of Natural Language Interfaces for Databases
- Prompting Generative AI with Interaction-Augmented Instructions
- SycEval: Evaluating LLM Sycophancy
- Augmenting a Large Language Model with a Combination of Text and Visual Data for Conversational Visualization of Global Geospatial Data
- 3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer
- Alignment faking in large language models
- MapIO: Embodied Interaction for the Accessibility of Tactile Maps Through Augmented Touch Exploration and Conversation
- VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format
- Personalization of Large Language Models: A Survey
- Word2Wave: Language Driven Mission Programming for Efficient Subsea Deployments of Marine Robots
- Visualizationary: Automating Design Feedback for Visualization Designers using LLMs
- Collaborative Forensic Autopsy Documentation and Supervised Report Generation Using a Hybrid Mixed-Reality Environment and Generative AI
- Generating Analytic Specifications for Data Visualization from Natural Language Queries using Large Language Models
- LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs
- Graph Retrieval-Augmented Generation: A Survey
- Voicing Uncertainty: How Speech, Text, and Visualizations Influence Decisions with Data Uncertainty
- How Aligned are Human Chart Takeaways and LLM Predictions? A Case Study on Bar Charts with Varying Layouts
- Can GPT-4 Models Detect Misleading Visualizations?
- Talk to the Wall: The Role of Speech Interaction in Collaborative Visual Analytics
- The State of Reproducibility Stamps for Visualization Research Papers
- Fine-Tuned Large Language Model for Visualization System: A Study on Self-Regulated Learning in Education
- Beyond Metrics: A Critical Analysis of the Variability in Large Language Model Evaluation Frameworks
- How Good (Or Bad) Are LLMs at Detecting Misleading Visualizations?
- Trust Your Gut: Comparing Human and Machine Inference from Noisy Visualizations
- A Survey on Employing Large Language Models for Text-to-SQL Tasks
- The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism
- Naturally Occurring Feedback is Common, Extractable and Useful
- VisEval: A Benchmark for Data Visualization in the Era of Large Language Models
- VideoLLM-online: Online Video Large Language Model for Streaming Video
- Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL
- Mixture-of-Agents Enhances Large Language Model Capabilities
- Save It for the "Hot" Day: An LLM-Empowered Visual Analytics System for Heat Risk Management
- CityGPT: Towards Urban IoT Learning, Analysis and Interaction with Multi-Agent System
- A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
- Leveraging Large Language Models to Enhance Domain Expert Inclusion in Data Science Workflows
- A Survey on the Memory Mechanism of Large Language Model based Agents
- GENEVIC: GENetic data Exploration and Visualization via Intelligent interactive Console
- InsightLens: Augmenting LLM-Powered Data Analysis with Interactive Insight Management and Navigation
- A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)
- ST-LLM: Large Language Models Are Effective Temporal Learners
- HINTs: Sensemaking on large collections of documents with Hypergraph visualization and INTelligent agents
- Retrieval-Augmented Generation for AI-Generated Content: A Survey
- Visualization Generation with Large Language Models: An Evaluation
- LSDvis: Hallucinatory Data Visualisations in Real World Environments
- Which is better? Exploring Prompting Strategy For LLM-based Metrics
- What's "up" with vision-language models? Investigating their struggle with spatial reasoning
- Natural Language Interfaces for Tabular Data Querying and Visualization: A Survey
- Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
- How Good is ChatGPT in Giving Advice on Your Visualization Design?
- VizAbility: Enhancing Chart Accessibility with LLM-based Conversational Interaction
- MemGPT: Towards LLMs as Operating Systems
- Data Formulator: AI-powered Concept-driven Visualization Authoring
- Talking to Data Visualizations: Opportunities and Challenges
- AI Deception: A Survey of Examples, Risks, and Potential Solutions
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- InkSight: Leveraging Sketch Interaction for Documenting Chart Findings in Computational Notebooks
- A Survey on Evaluation of Large Language Models
- A Survey on Evaluation of Large Language Models
- VisText: A Benchmark for Semantically Rich Chart Captioning
- AudioPaLM: A Large Language Model That Can Speak and Listen
- Autonomous GIS: the next-generation AI-powered GIS
- VOICE: Visual Oracle for Interaction, Conversation, and Explanation
- The Rational Agent Benchmark for Data Visualization
- A Survey of Large Language Models
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- GPT-4 Technical Report
- LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models
- Evaluating the Text-to-SQL Capabilities of Large Language Models
- Survey of Hallucination in Natural Language Generation
- BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
- An Evaluation-Focused Framework for Visualization Recommendation Algorithms
- The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
- AI4VIS: Survey on Artificial Intelligence Approaches for Data Visualization
- Open science saves lives: lessons from the COVID-19 pandemic
- Threats of a replication crisis in empirical computer science
- Language Models are Few-Shot Learners
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Quda: Natural Language Queries for Visual Data Analytics
- Survey on Individual Differences in Visualization
- PlotQA: Reasoning over Scientific Plots
- DVQA: Understanding Data Visualizations via Question Answering
- FigureQA: An Annotated Figure Dataset for Visual Reasoning
- Attention Is All You Need
- 3D-LLM: Injecting the 3D World into Large Language Models
Discussions
Related