Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability

2025/06/02 by Mengliang He, He, Mengliang, Zeng, Jiayi +10 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Model-Driven Software Engineering Techniques #Software Engineering (cs.SE) #Software Engineering Research #Software Testing and Debugging Techniques

paper · pdf · doi:10.48550/arxiv.2506.02073

openalex publication_date 2025/06/02 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

While large language models (LLMs) show promise in code generation, existing benchmarks neglect the flowchart-based code generation. To promote further research on flowchart-based code generation, this work presents Flow2Code, a novel benchmark for flowchart-based code generation evaluation. The evaluation dataset spans 15 programming languages and includes 5,622 code segments paired with 16,866 flowcharts of three types: code, UML, and pseudocode. Extensive experiments with 13 multimodal LLMs reveal that current LLMs can not generate code based on flowcharts perfectly. Besides, experiment results show that the supervised fine-tuning technique contributes greatly to the models' performance. We publicly release our code and datasets at https://github.com/hml-github/Flow2Code.

Citations

• DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
• An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• LLaVA-OneVision: Easy Visual Task Transfer
• MiniCPM-V: A GPT-4V Level MLLM on Your Phone
• DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
• MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems
• HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation
• CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
• MMBench: Benchmarking End-to-End Multi-modal DNNs and Understanding Their Hardware-Software Implications
• DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
• Multi-lingual Evaluation of Code Generation Models
• Program Synthesis with Large Language Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Measuring Coding Challenge Competence With APPS
• Towards Conversational Recommendation over Multi-Type Dialogs
• SPoC: Search-based Pseudocode to Code
• HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks

Cited by

• RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models
• Refining Critical Thinking in LLM Code Generation: A Faulty Premise-based Evaluation Framework

Related

• A Survey on Evaluating Large Language Models in Code Generation Tasks
• Large Language Models Meet NL2Code: A Survey
• Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications
• First Multi-Dimensional Evaluation of Flowchart Comprehension for Multimodal Large Language Models
• L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models