A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
2023/02/08 by Yejin Bang, Samuel Cahyawijaya, Bang, Yejin +23 · 2 voices · 358 citations
Computer Science · #Artificial intelligence #Automatic summarization #Computer science #Machine learning #Natural Language Processing Techniques #Natural language processing #Programming language #Semantic reasoner #Set (abstract data type) #Text Readability and Simplification #Topic Modeling #cs.AI #cs.CL
paper · pdf · doi:10.48550/arxiv.2302.04023
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/02/08 · openalex created_date 2023/02/11 · openalex updated_date 2026/07/28
Abstract
This paper proposes a framework for quantitatively evaluating interactive LLMs such as ChatGPT using publicly available data sets. We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. We find that ChatGPT outperforms LLMs with zero-shot learning on most tasks and even outperforms fine-tuned models on some tasks. We find that it is better at understanding non-Latin script languages than generating them. It is able to generate multimodal content from textual prompts, via an intermediate code generation step. Moreover, we find that ChatGPT is 63.41% accurate on average in 10 different reasoning categories under logical reasoning, non-textual reasoning, and commonsense reasoning, hence making it an unreliable reasoner. It is, for example, better at deductive than inductive reasoning. ChatGPT suffers from hallucination problems like other LLMs and it generates more extrinsic hallucinations from its parametric memory as it does not have access to an external knowledge base. Finally, the interactive feature of ChatGPT enables human collaboration with the underlying LLM to improve its performance, i.e, 8% ROUGE-1 on summarization and 2% ChrF++ on machine translation, in a multi-turn "prompt engineering" fashion. We also release codebase for evaluation set extraction.
Cited by
- ConsistencyGate: Preventing Memory Contamination in LLM Agents via Self-Consistency Admission Control
- LLMs on Drugs: Language Models Are Few-Shot Consumers
- Introducing ORKG ASK: an AI-driven Scholarly Literature Search and Exploration System Taking a Neuro-Symbolic Approach
- Bounty Hunter: Autonomous, Comprehensive Emulation of Multi-Faceted Adversaries
- Understanding LLM Agent Behaviours via Game Theory: Strategy Recognition, Biases and Multi-Agent Dynamics
- What Helps Language Models Predict Human Beliefs: Demographics or Prior Stances?
- MACEval: A Multi-Agent Continual Evaluation Network for Large Models
- HalluClean: A Unified Framework to Combat Hallucinations in LLMs
- An In-depth Study of LLM Contributions to the Bin Packing Problem
- HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placement
- MERIT: Modular Framework for Multimodal Misinformation Detection with Web-Grounded Reasoning
- TRUSTVIS: A Multi-Dimensional Trustworthiness Evaluation Framework for Large Language Models
- A Survey on Evaluation of Large Language Models
- How Well Can Preference Optimization Generalize Under Noisy Feedback?
- Automated Skill Decomposition Meets Expert Ontologies: Bridging the Granularity Gap with LLMs
- DACIP-RC: Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension on Business Conversations
- LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilization
- Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptation
- ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards
- Defeating Cerberus: Concept-Guided Privacy-Leakage Mitigation in Multimodal Language Models
- Large Language Models for Toxic Language Detection in Low-Resource Balkan Languages
- Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions
- Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment
- ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation
- Review of Hallucination Understanding in Large Language and Vision Models
- Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks
- Embedding Domain Knowledge for Large Language Models via Reinforcement Learning from Augmented Generation
- USB-Rec: An Effective Framework for Improving Conversational Recommendation Capability of Large Language Model
- Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
- Rationality Check! Benchmarking the Rationality of Large Language Models
- Exploring Data and Parameter Efficient Strategies for Arabic Dialect Identifications
- Natural Language Satisfiability: Exploring the Problem Distribution and Evaluating Transformer-based Language Models
- The imitation game: Detecting human and AI-generated texts in the era of ChatGPT and BARD
- Can Smaller LLMs do better? Unlocking Cross-Domain Potential through Parameter-Efficient Fine-Tuning for Text Summarization
- KG-CQR: Leveraging Structured Relation Representations in Knowledge Graphs for Contextual Query Retrieval
- Charting the Future of Scholarly Knowledge with AI: A Community Perspective
- Online Anti-sexist Speech: Identifying Resistance to Gender Bias in Political Discourse
- Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends
- January Food Benchmark (JFB): A Public Benchmark Dataset and Evaluation Suite for Multimodal Food Analysis
- SAEMark: Steering Personalized Multilingual LLM Watermarks with Sparse Autoencoders
- Guided Navigation in Knowledge-Dense Environments: Structured Semantic Exploration with Guidance Graphs
- PromptAL: Sample-Aware Dynamic Soft Prompts for Few-Shot Active Learning
- Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models
- Out-of-Context Abduction: LLMs Make Inferences About Procedural Data Leveraging Declarative Facts in Earlier Training Data
- What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representations
- InvestAlign: Overcoming Data Scarcity in Aligning Large Language Models with Investor Decision-Making Processes under Herd Behavior
- WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
- Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models
- Holistic Artificial Intelligence in Medicine; improved performance and explainability
- Embodied AI Agents: Modeling the World
- Weak-to-Strong GraphRAG: Aligning Weak Retrievers with Large Language Models for Graph-based Retrieval Augmented Generation
- PDF Retrieval Augmented Question Answering
- Mapping (A)Ideology: A Taxonomy of European Parties Using Generative LLMs as Zero-Shot Learners
- Causes in neuron diagrams, and testing causal reasoning in Large Language Models. A glimpse of the future of philosophy?
- A Culturally-diverse Multilingual Multimodal Video Benchmark & Model
- PCoT: Persuasion-Augmented Chain of Thought for Detecting Fake News and Social Media Disinformation
- Fake Friends and Sponsored Ads: The Risks of Advertising in Conversational Search
- The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs
- Can Theoretical Physics Research Benefit from Language Agents?
- RewardAnything: Generalizable Principle-Following Reward Models
- Assessing and Enhancing the Robustness of LLM-based Multi-Agent Systems Through Chaos Engineering
- CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstention
- Optimizing Question Semantic Space for Dynamic Retrieval-Augmented Multi-hop Question Answering
- Event-aware analysis of cross-city visitor flows using large language models and social media data
- MMTBENCH: A Unified Benchmark for Complex Multimodal Table Reasoning
- A Lightweight Multi-Expert Generative Language Model System for Engineering Information and Knowledge Extraction
- SpokenNativQA: Multilingual Everyday Spoken Queries for LLMs
- Multilingual Question Answering in Low-Resource Settings: A Dzongkha-English Benchmark for Foundation Models
- Debate-to-Detect: Reformulating Misinformation Detection as a Real-World Debate with Large Language Models
- From Reasoning to Generalization: Knowledge-Augmented LLMs for ARC Benchmark
- HASH-RAG: Bridging Deep Hashing with Retriever for Efficient, Fine Retrieval and Augmented Generation
- UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking
- Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes
- The Graph Language: How Knowledge Graphs Speak to Large Language Models
- BugRepro: Enhancing Android Bug Reproduction with Domain-Specific Knowledge Integration
- Auditing Meta-Cognitive Hallucinations in Reasoning Large Language Models
- Latent Veracity Inference for Identifying Errors in Stepwise Reasoning
- Extracting Visual Plans from Unlabeled Videos via Symbolic Guidance
- Towards Contamination Resistant Benchmarks
- ”My AI is Lying to Me”: User-reported LLM hallucinations in AI mobile apps reviews
- A Survey on Privacy Risks and Protection in Large Language Models
- Spilled Energy in Large Language Models
- Low-Cost Black-Box Detection of LLM Hallucinations via Dynamical System Prediction
- Can We Enhance Bug Report Quality Using LLMs?: An Empirical Study of LLM-Based Bug Report Generation
- Evaluating ChatGPT on Medical Information Extraction Tasks: Performance, Explainability and Beyond
- HalluLens: LLM Hallucination Benchmark
- Interpretable Locomotion Prediction in Construction Using a Memory-Driven LLM Agent With Chain-of-Thought Reasoning
- CRAVE: A Conflicting Reasoning Approach for Explainable Claim Verification Using LLMs
- Simulating Before Planning: Constructing Intrinsic User World Model for User-Tailored Dialogue Policy Planning
- CRAB: A Benchmark for Evaluating Curation of Retrieval-Augmented LLMs in Biomedicine
- C-FAITH: A Chinese Fine-Grained Benchmark for Automated Hallucination Evaluation
- How to Detect and Defeat Molecular Mirage: A Metric-Driven Benchmark for Hallucination in LLM-based Molecular Comprehension
- Bypassing Safety Guardrails in LLMs Using Humor
- COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for Alignment with Human Values
- Stock Market Forecasting: From Traditional Predictive Models to Large Language Models
Discussions
Related