Evaluating LLMs with Multiple Problems at once

2024/06/16 by Zhengxiang Wang, Wang, Zhengxiang, Jordan Kodner +3 · 2 citations
Computer Science · #Library Science and Information Systems

paper · pdf · doi:10.48550/arxiv.2406.10786

Abstract

This paper shows the benefits and fruitfulness of evaluating LLMs with multiple problems at once, a paradigm we call multi-problem evaluation (MPE). Unlike conventional single-problem evaluation, where a prompt presents a single problem and expects one specific answer, MPE places multiple problems together in a single prompt and assesses how well an LLM answers all these problems in a single output. Leveraging 6 classification and 12 reasoning benchmarks that already exist, we introduce a new benchmark called ZeMPE (Zero-shot Multi-Problem Evaluation), comprising 53,100 zero-shot multi-problem prompts. We experiment with a total of 13 LLMs from 5 model families on ZeMPE to present a comprehensive and systematic MPE. Our results show that LLMs are capable of handling multiple problems from a single data source as well as handling them separately, but there are conditions this multiple problem handling capability falls short. In addition, we perform in-depth further analyses and explore model-level factors that may enable multiple problem handling capabilities in LLMs. We release our corpus and code to facilitate future research.

Cited by

• REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once
• RWESummary: A Framework and Test for Choosing Large Language Models to Summarize Real-World Evidence (RWE) Studies

Related

• State of What Art? A Call for Multi-Prompt LLM Evaluation
• Multiple-Choice Questions are Efficient and Robust LLM Evaluators
• Relay-Bench: Evaluating LLMs on Multi-Domain Reasoning Chains
• How to Get Your LLM to Generate Challenging Problems for Evaluation
• AQA-Bench: An Interactive Benchmark for Evaluating LLMs' Sequential Reasoning Ability