Multimedia and Visual Analytics in the Agentic Era

2025/04/08 by Marcel Worring, Jan Zahálka, Worring, Marcel +7 · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Data Visualization and Analytics #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Mobile Crowdsensing and Crowdsourcing #Multimedia (cs.MM) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2504.06138

openalex publication_date 2025/04/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Professional users need tools to help them gain actionable insights from large multimedia collections. Foundation models and AI agents have rapidly changed the playing field, and improving their accuracy, trustworthiness, and reasoning capabilities are active topics in the computer vision, machine learning, and multimedia communities. Most current research focuses on benchmark driven algorithmic improvements. The multimedia community is the place to go beyond algorithms and consider complete multimedia analytics systems that support professional users in their complex tasks and achieve a true teaming of humans and AI. Supporting users with machine learning and visualizations has been studied for decades in the visual analytics field. In this paper, we propose a framework to bring multimedia and visual analytics together and indicate how it could impact current and new multimedia analytics solutions. Additional information can be found at https://staff.fnwi.uva.nl/m.worring/analytics-model.html

Citations

• DiLLS: Interactive Diagnosis of LLM-based Multi-agent Systems via Layered Summary of Agent Behaviors
• Interactive Hypergraph Visual Analytics for Exploring Large and Complex Image Collections
• The Human-Data-Model Interaction Canvas for Visual Analytics
• ArtRAG: Retrieval-Augmented Generation with Structured Context for Visual Art Understanding
• Towards Temporal-Aware Multi-Modal Retrieval Augmented Generation in Finance
• Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
• ContextCite: Attributing Model Generation to Context
• The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
• GenLens: A Systematic Evaluation of Visual GenAI Model Outputs
• Demystifying Chains, Trees, and Graphs of Thoughts
• MULTI-CASE: A Transformer-based Ethics-aware Multimodal Investigative Intelligence Framework
• Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
• Levels of AGI for Operationalizing Progress on the Path to AGI
• Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark
• The Rise and Potential of Large Language Model Based Agents: A Survey
• Interactive video retrieval in the age of effective joint embedding deep models: lessons from the 11th VBS
• RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization
• Lost in the Middle: How Language Models Use Long Contexts
• CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
• Human-Computer Collaboration for Visual Analytics: an Agent-based Framework
• How Does Attention Work in Vision Transformers? A Visual Analytics Attempt
• Retrieving Multimodal Information for Augmented Generation: A Survey
• Large Language Models are Zero-Shot Reasoners
• MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning
• Promoting Ethical Awareness in Communication Analysis: Investigating Potentials and Limits of Visual Analytics for Intelligence Applications
• Training language models to follow instructions with human feedback
• Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models
• VideoModerator: A Risk-aware Framework for Multimodal Video Moderation in E-Commerce
• A Survey on Bias in Visual Datasets
• Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• Learning to summarize from human feedback
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Recent Research Advances on Interactive Machine Learning
• Vega-Lite: A Grammar of Interactive Graphics
• Attention Is All You Need
• ELIZA—a computer program for the study of natural language communication between man and machine
• Retrieval-Augmented Multimodal Language Modeling

Cited by

• Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI

Related

• Human-Computer Collaboration for Visual Analytics: an Agent-based Framework
• The Human-Data-Model Interaction Canvas for Visual Analytics
• A Design Space for Intelligent Agents in Mixed-Initiative Visual Analytics
• AVA: Towards Agentic Video Analytics with Vision Language Models
• Visual Analytics Challenges and Trends in the Age of AI: The BigVis Community Perspective