Adaptively profiling models with task elicitation

2025/03/03 by Brown, Davis, Balehannina, Prithvi, Jin, Helen +3 · 1 citation
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2503.01986

Abstract

Language model evaluations often fail to characterize consequential failure modes, forcing experts to inspect outputs and build new benchmarks. We introduce task elicitation, a method that automatically builds new evaluations to profile model behavior. Task elicitation finds hundreds of natural-language tasks -- an order of magnitude more than prior work -- where frontier models exhibit systematic failures, in domains ranging from forecasting to online harassment. For example, we find that Sonnet 3.5 over-associates quantum computing and AGI and that o3-mini is prone to hallucination when fabrications are repeated in-context.

Cited by

• BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks

Related

• TaskEval: Synthesised Evaluation for Foundation-Model Tasks
• Eliciting Language Model Behaviors with Investigator Agents
• Training on the Test Task Confounds Evaluation and Emergence
• Task Contamination: Language Models May Not Be Few-Shot Anymore
• Investigating task-specific prompts and sparse autoencoders for activation monitoring