SciCap: Generating Captions for Scientific Figures

2021/10/22 by Hsu, Ting-Yao, Giles, C. Lee, Huang, Ting-Hao 'Kenneth' · 13 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2110.11624

Abstract

Researchers use figures to communicate rich, complex information in scientific papers. The captions of these figures are critical to conveying effective messages. However, low-quality figure captions commonly occur in scientific articles and may decrease understanding. In this paper, we propose an end-to-end neural framework to automatically generate informative, high-quality captions for scientific figures. To this end, we introduce SCICAP, a large-scale figure-caption dataset based on computer science arXiv papers published between 2010 and 2020. After pre-processing - including figure-type classification, sub-figure identification, text normalization, and caption text selection - SCICAP contained more than two million figures extracted from over 290,000 papers. We then established baseline models that caption graph plots, the dominant (19.2%) figure type. The experimental results showed both opportunities and steep challenges of generating captions for scientific figures.

Cited by

• Five Years of SciCap: What We Learned and Future Directions for Scientific Figure Captioning
• CycleChart: A Unified Consistency-Based Learning Framework for Bidirectional Chart Understanding and Generation
• Exploring LLMs for Scientific Information Extraction Using The SciEx Framework
• Charts Are Not Images: On the Challenges of Scientific Chart Editing
• Operon: Incremental Construction of Ragged Data via Named Dimensions
• Leveraging Author-Specific Context for Scientific Figure Caption Generation: 3rd SciCap Challenge
• QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding
• Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterization
• ChartCap: Mitigating Hallucination of Dense Chart Captioning
• A Survey of AI for Materials Science: Foundation Models, LLM Agents, Datasets, and Tools
• LaMP-Cap: Personalized Figure Caption Generation With Multimodal Figure Profiles
• MuSciClaims: Multimodal Scientific Claim Verification
• ChartLens: Fine-grained Visual Attribution in Charts

Related

• SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning
• Figuring out Figures: Using Textual References to Caption Scientific Figures
• Five Years of SciCap: What We Learned and Future Directions for Scientific Figure Captioning
• Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023
• Multi-LLM Collaborative Caption Generation in Scientific Documents