2025/05/16 by Banca Calvo Figueras, Figueras, Banca Calvo, Rodrigo Agerri +1 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multi-Agent Systems and Negotiation #Sentiment Analysis and Opinion Mining #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2505.11341
openalex publication_date 2025/05/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The task of Critical Questions Generation (CQs-Gen) aims to foster critical thinking by enabling systems to generate questions that expose underlying assumptions and challenge the validity of argumentative reasoning structures. Despite growing interest in this area, progress has been hindered by the lack of suitable datasets and automatic evaluation standards. This paper presents a comprehensive approach to support the development and benchmarking of systems for this task. We construct the first large-scale dataset including ~5K manually annotated questions. We also investigate automatic evaluation methods and propose reference-based techniques as the strategy that best correlates with human judgments. Our zero-shot evaluation of 11 LLMs establishes a strong baseline while showcasing the difficulty of the task. Data and code plus a public leaderboard are provided to encourage further research, not only in terms of model performance, but also to explore the practical benefits of CQs-Gen for both automated reasoning and human critical thinking.