CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Model

2025/06/16 by Jiangtong Li, Li, Jiangtong, Yiyun Zhu +7
Computer Science · Decision Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Mathematics, Computing, and Information Processing #Stock Market Forecasting Methods #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.13055

openalex publication_date 2025/06/16 · openalex created_date 2025/10/13 · openalex updated_date 2026/07/28

Abstract

Multimodal Large Language Models (MLLMs) have rapidly evolved with the growth of Large Language Models (LLMs) and are now applied in various fields. In finance, the integration of diverse modalities such as text, charts, and tables is crucial for accurate and efficient decision-making. Therefore, an effective evaluation system that incorporates these data types is essential for advancing financial application. In this paper, we introduce CFBenchmark-MM, a Chinese multimodal financial benchmark with over 9,000 image-question pairs featuring tables, histogram charts, line charts, pie charts, and structural diagrams. Additionally, we develop a staged evaluation system to assess MLLMs in handling multimodal information by providing different visual content step by step. Despite MLLMs having inherent financial knowledge, experimental results still show limited efficiency and robustness in handling multimodal financial context. Further analysis on incorrect responses reveals the misinterpretation of visual content and the misunderstanding of financial concepts are the primary issues. Our research validates the significant, yet underexploited, potential of MLLMs in financial analysis, highlighting the need for further development and domain-specific optimization to encourage the enhanced use in financial domain.

Citations

• MME-Industry: A Cross-Industry Multimodal Evaluation Benchmark
• INVESTORBENCH: A Benchmark for Financial Decision-Making Tasks with LLM-based Agent
• SuperCLUE-Fin: Graded Fine-Grained Analysis of Chinese LLMs on Diverse Financial Tasks and Applications
• FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models
• InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
• mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
• DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning
• MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
• MMBench: Is Your Multi-modal Model an All-around Player?
• QLoRA: Efficient Finetuning of Quantized LLMs
• XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• BloombergGPT: A Large Language Model for Finance
• LLaMA: Open and Efficient Foundation Language Models
• BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark
• VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
• FinQA: A Dataset of Numerical Reasoning over Financial Data
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
• GLM: General Language Model Pretraining with Autoregressive Blank Infilling
• Learning Transferable Visual Models From Natural Language Supervision
• Language Models are Few-Shot Learners
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• XLNet: Generalized Autoregressive Pretraining for Language Understanding

Related

• MME-Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning
• CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model
• VisFinEval: A Scenario-Driven Chinese Multimodal Benchmark for Holistic Financial Understanding
• MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application
• FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challenging