Benchmarks as Microscopes: A Call for Model Metrology

2024/07/22 by Michael Saxon, Ari Holtzman, Saxon, Michael +7 · 2 voices · 8 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Software Engineering (cs.SE) #cs.CL #cs.SE

paper · pdf · doi:10.48550/arxiv.2407.16711

arxiv published 2024/07/22 · arxiv updated 2024/07/30

Abstract

Modern language models (LMs) pose a new challenge in capability assessment. Static benchmarks inevitably saturate without providing confidence in the deployment tolerances of LM-based systems, but developers nonetheless claim that their models have generalized traits such as reasoning or open-domain language understanding based on these flawed metrics. The science and practice of LMs requires a new approach to benchmarking which measures specific capabilities with dynamic assessments. To be confident in our metrics, we need a new discipline of model metrology -- one which focuses on how to generate benchmarks that predict performance under deployment. Motivated by our evaluation criteria, we outline how building a community of model metrology practitioners -- one focused on building tools and studying how to measure system capabilities -- is the best way to meet these needs to and add clarity to the AI discussion.

Cited by

• Rigorous Interpretation Is a Form of Evaluation
• Can They Dixit? Yes they Can! Dixit as a Playground for Multimodal Language Model Capabilities
• How Reliable is Language Model Micro-Benchmarking?
• Towards Ecologically Valid LLM Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitioners
• Toward a Theory of Generalizability in LLM Mechanistic Interpretability Research
• A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users
• Fluid Language Model Benchmarking
• Culture is Everywhere: A Call for Intentionally Cultural Evaluation

Discussions

• 🤖 When software ate the world, pentesting became a profession outside infosec conferences. When telescopes got complex, metrologists replaced astronomers’ glassmaking hobby. Our COLM paper asks: is i [bsky, 3 points, 0 comments]
• Much of my work touches on themes of *making serious technical contributions to enable more rigorous evaluation*. For a deep dive into why this direction is impt, how to do it, and the kinds of futur [bsky, 0 points, 1 comments]

Related

• Metrology for AI: From Benchmarks to Instruments
• Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks
• RepBench: Compiling Benchmarks into Capability Representations for Large Language Models
• Beyond Benchmarking: A New Paradigm for Evaluation and Assessment of Large Language Models
• A Survey on Large Language Model Benchmarks