RECODE: Reasoning Through Code Generation for Visual Question Answering

2025/10/15 by Jun-Hong Shen, Shen, Junhong, Mu Cai +11
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.13756

openalex publication_date 2025/10/15 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/28

Abstract

Multimodal Large Language Models (MLLMs) struggle with precise reasoning for structured visuals like charts and diagrams, as pixel-based perception lacks a mechanism for verification. To address this, we propose to leverage derendering -- the process of reverse-engineering visuals into executable code -- as a new modality for verifiable visual reasoning. Specifically, we propose RECODE, an agentic framework that first generates multiple candidate programs to reproduce the input image. It then uses a critic to select the most faithful reconstruction and iteratively refines the code. This process not only transforms an ambiguous perceptual task into a verifiable, symbolic problem, but also enables precise calculations and logical inferences later on. On various visual reasoning benchmarks such as CharXiv, ChartQA, and Geometry3K, RECODE significantly outperforms methods that do not leverage code or only use code for drawing auxiliary lines or cropping. Our work demonstrates that grounding visual perception in executable code provides a new path toward more accurate and verifiable multimodal reasoning.

Citations

• Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction
• SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning
• VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selection
• GRIT: Teaching MLLMs to Think with Images
• Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs
• DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning
• ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
• CodePDE: An Inference Framework for LLM-driven PDE Solver Generation
• Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation
• Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity
• ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• On Pre-training of Multimodal Language Models Customized for Chart Understanding
• CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs
• Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models
• Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning
• UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation
• LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
• UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning
• Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings
• Tool Learning with Foundation Models
• ViperGPT: Visual Inference via Python Execution for Reasoning
• Cross-Modal Fine-Tuning: Align then Refine
• Toolformer: Language Models Can Teach Themselves to Use Tools
• DePlot: One-shot visual language reasoning by plot-to-table translation
• MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering
• Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks
• Visual Programming: Compositional visual reasoning without training
• Efficient Architecture Search for Diverse Tasks
• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
• PlotQA: Reasoning over Scientific Plots
• Learning to Infer Graphics Programs from Hand-Drawn Images

Related

• VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation
• Decoupling Reasoning and Perception: An LLM-LMM Framework for Faithful Visual Reasoning
• CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoning
• Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback
• From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesis