SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark

2024/02/06 by Zhenwen Liang, Kehan Guo, Liang, Zhenwen +17 · 7 citations
Computer Science · Psychology · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Educational Technology and Assessment #FOS: Computer and information sciences #Innovative Teaching and Learning Methods #Online and Blended Learning

paper · pdf · doi:10.48550/arxiv.2402.05138

openalex publication_date 2024/02/06 · openalex created_date 2024/02/10 · openalex updated_date 2026/07/28

Abstract

The paper introduces SceMQA, a novel benchmark for scientific multimodal question answering at the college entrance level. It addresses a critical educational phase often overlooked in existing benchmarks, spanning high school to pre-college levels. SceMQA focuses on core science subjects including Mathematics, Physics, Chemistry, and Biology. It features a blend of multiple-choice and free-response formats, ensuring a comprehensive evaluation of AI models' abilities. Additionally, our benchmark provides specific knowledge points for each problem and detailed explanations for each answer. SceMQA also uniquely presents problems with identical contexts but varied questions to facilitate a more thorough and accurate assessment of reasoning capabilities. In the experiment, we evaluate both open-source and close-source state-of-the-art Multimodal Large Language Models (MLLMs), across various experimental settings. The results show that further research and development are needed in developing more capable MLLM, as highlighted by only 50% to 60% accuracy achieved by the strongest models. Our benchmark and analysis will be available at https://scemqa.github.io/

Cited by

• ALDEN: Reinforcement Learning for Active Navigation and Evidence Gathering in Long Documents
• ViSurf: Visual Supervised-and-Reinforcement Fine-Tuning for Large Vision-and-Language Models
• Self-Improvement in Multimodal Large Language Models: A Survey
• Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards
• GRAFT: GRaPH and Table Reasoning for Textual Alignment -- A Benchmark for Structured Instruction Following and Visual Reasoning
• StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Models
• QCBench: Evaluating Large Language Models on Domain-Specific Quantitative Chemistry

Related

• MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
• EduVidQA: Generating and Evaluating Long-form Answers to Student Questions based on Lecture Videos
• VisScience: An Extensive Benchmark for Evaluating K12 Educational Multi-modal Scientific Reasoning
• Automated MCQA Benchmarking at Scale: Evaluating Reasoning Traces as Retrieval Sources for Domain Adaptation of Small Language Models
• MultiModalQA: Complex Question Answering over Text, Tables and Images