2022/06/28 by Victor Vikram Odouard, Melanie Mitchell, Odouard, Victor Vikram +1 · 2 citations
Computer Science · #AI-based Problem Solving and Planning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Semantic Web and Ontologies #Topic Modeling #cs.AI #cs.LG
paper · pdf · doi:10.48550/arxiv.2206.14187
EBeM'22: AI Evaluation Beyond Metrics, July 24, 2022, Vienna, Austria
arxiv created 2022/06/28 · openalex publication_date 2022/06/28 · arxiv updated 2022/06/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
A long-held objective in AI is to build systems that understand concepts in a humanlike way. Setting aside the difficulty of building such a system, even trying to evaluate one is a challenge, due to present-day AI's relative opacity and its proclivity for finding shortcut solutions. This is exacerbated by humans' tendency to anthropomorphize, assuming that a system that can recognize one instance of a concept must also understand other instances, as a human would. In this paper, we argue that understanding a concept requires the ability to use it in varied contexts. Accordingly, we propose systematic evaluations centered around concepts, by probing a system's ability to use a given concept in many different instantiations. We present case studies of such an evaluations on two domains -- RAVEN (inspired by Raven's Progressive Matrices) and the Abstraction and Reasoning Corpus (ARC) -- that have been used to develop and assess abstraction abilities in AI systems. Our concept-based approach to evaluation reveals information about AI systems that conventional test sets would have left hidden.