Assessing GPT4-V on Structured Reasoning Tasks

2023/12/13 by Singh, Mukul, Cambronero, José, Gulwani, Sumit +2 · 4 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2312.11524

Abstract

Multi-modality promises to unlock further uses for large language models. Recently, the state-of-the-art language model GPT-4 was enhanced with vision capabilities. We carry out a prompting evaluation of GPT-4V and five other baselines on structured reasoning tasks, such as mathematical reasoning, visual data analysis, and code generation. We show that visual Chain-of-Thought, an extension of Chain-of-Thought to multi-modal LLMs, yields significant improvements over the vanilla model. We also present a categorized analysis of scenarios where these models perform well and where they struggle, highlighting challenges associated with coherent multimodal reasoning.

Cited by

• MarineEval: Assessing the Marine Intelligence of Vision-Language Models
• MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models
• AgroCoT: A Chain-of-Thought Benchmark for Evaluating Reasoning in Vision-Language Models for Agriculture
• Score the Steps, Not Just the Goal: VLM-Based Subgoal Evaluation for Robotic Manipulation

Related

• CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models
• RBench-V: A Primary Assessment for Visual Reasoning Models with Multi-modal Outputs
• Skywork R1V: Pioneering Multimodal Reasoning with Chain-of-Thought
• GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinking
• VGR: Visual Grounded Reasoning