A Diagram Is Worth A Dozen Images
2016/03/24 by Aniruddha Kembhavi, Kembhavi, Aniruddha, Mike Salvato +9 · 174 citations
Computer Science · #Data Visualization and Analytics #Multimodal Machine Learning Applications #Video Analysis and Summarization
paper · pdf · doi:10.48550/arxiv.1603.07396
Abstract
Diagrams are common tools for representing complex concepts, relationships and events, often when it would be difficult to portray the same information with natural images. Understanding natural images has been extensively studied in computer vision, while diagram understanding has received little attention. In this paper, we study the problem of diagram interpretation and reasoning, the challenging task of identifying the structure of a diagram and the semantics of its constituents and their relationships. We introduce Diagram Parse Graphs (DPG) as our representation to model the structure of diagrams. We define syntactic parsing of diagrams as learning to infer DPGs for diagrams and study semantic interpretation and reasoning of diagrams in the context of diagram question answering. We devise an LSTM-based method for syntactic parsing of diagrams and introduce a DPG-based attention model for diagram question answering. We compile a new dataset of diagrams with exhaustive annotations of constituents and relationships for over 5,000 diagrams and 15,000 questions and answers. Our results show the significance of our models for syntactic parsing and question answering in diagrams using DPGs.
Citations
Cited by
- DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes
- Same or Not? Enhancing Visual Perception in Vision-Language Models
- VL-RouterBench: A Benchmark for Vision-Language Model Routing
- Self-Rewarded Multimodal Coherent Reasoning Across Diverse Visual Domains
- Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbone
- Masking Teacher and Reinforcing Student for Distilling Vision-Language Models
- Disentangling Semantic Attention from Structural Bias in the Attention Manifold
- Are Prompt Optimizers Blind? Cross-Modal Visual Feedback for Automatic Prompt Optimization
- UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models
- Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
- LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models
- D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning
- CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion
- Differences That Matter: Auditing Models for Capability Gap Discovery and Rectification
- Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
- DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models
- T5Gemma 2: Seeing, Reading, and Understanding Longer
- SDAR-VL: Stable and Efficient Block-wise Diffusion for Vision-Language Understanding
- HyperVL: An Efficient and Dynamic Multimodal Large Language Model for Edge Devices
- Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoning
- DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Model
- Image Tiling for High-Resolution Reasoning: Balancing Local Detail with Global Context
- Limits and Gains of Test-Time Scaling in Vision-Language Reasoning
- Towards Fine-Grained Recognition with Large Visual Language Models: Benchmark and Optimization Strategies
- InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
- Towards Lossless Ultimate Vision Token Compression for VLMs
- The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models
- Qwen3.5-Omni Technical Report
- Towards Cross-View Point Correspondence in Vision-Language Models
- Jina-VLM: Small Multilingual Vision Language Model
- OneThinker: All-in-one Reasoning Model for Image and Video
- MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigm
- Lost in Modality: Evaluating the Effectiveness of Text-Based Membership Inference Attacks on Large Multimodal Models
- VACoT: Rethinking Visual Data Augmentation with VLMs
- TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models
- Script: Graph-Structured and Query-Conditioned Semantic Token Pruning for Multimodal Large Language Models
- Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Models
- Visual Generation Tuning
- VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction
- From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images
- Qwen3-VL Technical Report
- EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokens
- Text-Guided Semantic Image Encoder
- LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight
- Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs
- INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Models
- OrdMoE: Preference Alignment via Hierarchical Expert Group Ranking in Multimodal Mixture-of-Experts LLMs
- Parallel Vision Token Scheduling for Fast and Accurate Multimodal LMMs Inference
- Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning
- FastMMoE: Accelerating Multimodal Large Language Models through Dynamic Expert Activation and Routing-Aware Token Pruning
- Beyond Multiple Choice: Verifiable OpenQA for Robust Vision-Language RFT
- EvoLMM: Self-Evolving Large Multimodal Models with Continuous Rewards
- Learning to Think Fast and Slow for Visual Language Models
- Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
- RedVTP: Training-Free Acceleration of Diffusion Vision-Language Models Inference via Masked Token-Guided Visual Token Pruning
- Viper-F1: Fast and Fine-Grained Multimodal Understanding with Cross-Modal State-Space Modulation
- The Ouroboros of Benchmarking: Reasoning Evaluation in an Era of Saturation
- CoLM: Collaborative Large Models via A Client-Server Paradigm
- Unveiling Modality Bias: Automated Sample-Specific Analysis for Multimodal Misinformation Benchmarks
- Visual Spatial Tuning
- Cambrian-S: Towards Spatial Supersensing in Video
- NVIDIA Nemotron Nano V2 VL
- From Pixels to Paths: A Multi-Agent Framework for Editable Scientific Illustration
- Dynamic Graph Generation Network: Generating Relational Knowledge from Diagrams
- Visual Question Answering: A Survey of Methods and Datasets
- Progressive Multimodal Alignment for Continual Instruction Tuning
- VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs
- Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation
- Revisiting Multimodal Positional Encoding in Vision-Language Models
- OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM
- Beyond Reasoning Gains: Mitigating General Capabilities Forgetting in Large Reasoning Models
- NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation
- ARGenSeg: Image Segmentation with Autoregressive Image Generation Model
- EmbodiedBrain: Expanding Performance Boundaries of Task Planning for Embodied Intelligence
- GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMs
- Metis-HOME: Hybrid Optimized Mixture-of-Experts for Multimodal Reasoning
- CARES: Context-Aware Resolution Selector for VLMs
- Unified Reinforcement and Imitation Learning for Vision-Language Models
- Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Images
- SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference
- FineVision: Open Data Is All You Need
- ZSPAPrune: Zero-Shot Prompt-Aware Token Pruning for Vision-Language Models
- Res-Bench: Benchmarking the Robustness of Multimodal Large Language Models to Dynamic Resolution Input
- MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Models
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale
- Vision-Centric Activation and Coordination for Multimodal Large Language Models
- InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue
- DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Search
- CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs
- MetaCaptioner: Towards Generalist Visual Captioning with Open-source Suites
- ViCO: A Training Strategy towards Semantic Aware Dynamic High-Resolution
- HoneyBee: Data Recipes for Vision-Language Reasoners
- FlexAC: Towards Flexible Control of Associative Reasoning in Multimodal Large Language Models
- MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understanding
- How to Teach Large Multimodal Models New Skills
- Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perception
- IntentionVLA: Generalizable and Efficient Embodied Intention Reasoning for Human-Robot Interaction
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints
- ImageNet-Think-250K: A Large-Scale Synthetic Dataset for Multimodal Reasoning for Vision Language Models
- TTRV: Test-Time Reinforcement Learning for Vision Language Models
- Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizer
- Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment
- Efficient Test-Time Scaling for Small Vision-Language Models
- MathSticks: A Benchmark for Visual Symbolic Compositional Reasoning with Matchstick Puzzles
- Apriel-1.5-15b-Thinker
- VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs
- Skip-It? Theoretical Conditions for Layer Skipping in Vision-Language Models
- PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applications
- RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks
- HIVTP: A Training-Free Method to Improve VLMs Efficiency via Hierarchical Visual Token Pruning Using Middle-Layer-Based Importance Score
- RIV: Recursive Introspection Mask Diffusion Vision Language Model
- Multi-modal Data Spectrum: Multi-modal Datasets are Multi-dimensional
- REMA: A Unified Reasoning Manifold Framework for Interpreting Large Language Model
- Chimera: Diagnosing Shortcut Learning in Visual-Language Understanding
- ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models
- MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources
- DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning
- Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models
- Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards
- OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignment
- Visual Instruction Pretraining for Domain-Specific Foundation Models
- Qwen3-Omni Technical Report
- MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
- Pyramid Token Pruning for High-Resolution Large Vision-Language Models via Region, Token, and Instruction-Guided Importance
- Qianfan-VL: Domain-Enhanced Universal Vision-Language Models
- Pointing to a Llama and Call it a Camel: On the Sycophancy of Multimodal Large Language Models
- MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
- AToken: A Unified Tokenizer for Vision
- Towards Rationale-Answer Alignment of LVLMs via Self-Rationale Calibration
- Explain Before You Answer: A Survey on Compositional Visual Reasoning
- MEENA (PersianMMMU): Multimodal-Multilingual Educational Exams for N-level Assessment
- The Art of Saying "Maybe": A Conformal Lens for Uncertainty Benchmarking in VLMs
- AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Models
- MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe
- Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Models
- 3D Aware Region Prompted Vision Language Model
- MindVL: Towards Efficient and Effective Training of Multimodal Large Language Models on Ascend NPUs
- Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Models
- VARCO-VISION-2.0 Technical Report
- DVQA: Understanding Data Visualizations via Question Answering
- DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Training
- Promptception: How Sensitive Are Large Multimodal Models to Prompts?
- OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation
- VLMs-in-the-Wild: Bridging the Gap Between Academic Benchmarks and Enterprise Reality
- MoPEQ: Mixture of Mixed Precision Quantized Experts
- Implicit Reasoning in Large Language Models: A Comprehensive Survey
- Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editing
- Improving Large Vision and Language Models by Learning from a Panel of Peers
- Kwai Keye-VL 1.5 Technical Report
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models
- LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Model
- Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safety
- CAD2DMD-SET: Synthetic Generation Tool of Digital Measurement Device CAD Model Datasets for fine-tuning Large Vision-Language Models
- R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learning
- Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models
- Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curation
- LangVision-LoRA-NAS: Neural Architecture Search for Variable LoRA Rank in Vision Language Models
- Ovis2.5 Technical Report
- BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning
- MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models
- MolmoAct: Action Reasoning Models that can Reason in Space
- MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark
- X-SAM: From Segment Anything to Any Segmentation
- Training-Free Multimodal Large Language Model Orchestration
- A Rolling Stone Gathers No Moss: Adaptive Policy Optimization for Stable Self-Evaluation in Large Multimodal Models
- Evaluating Variance in Visual Question Answering Benchmarks
- MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosis
- UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing
- VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning
- MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning
Related