2021/02/09 by Robyn Kozierok, John Aberdeen, Kozierok, Robyn +18 · 1 citation
Biochemistry, Genetics and Molecular Biology · Computer Science · Medicine · #AI in Service Interactions #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Biomedical Text Mining and Ontologies #Computation and Language (cs.CL) #FOS: Computer and information sciences #H.5.2 #Human-Computer Interaction (cs.HC) #I.2.11 #I.2.7 #Multiagent Systems (cs.MA) #Multimedia (cs.MM) #Software Engineering Techniques and Practices #Topic Modeling #cs.AI #cs.CL #cs.HC #cs.MA #cs.MM
paper · pdf · doi:10.48550/arxiv.2102.04958
20 pages, 21 figures
arxiv created 2021/02/09 · openalex publication_date 2021/02/09 · arxiv updated 2021/02/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
There is a growing desire to create computer systems that can communicate effectively to collaborate with humans on complex, open-ended activities. Assessing these systems presents significant challenges. We describe a framework for evaluating systems engaged in open-ended complex scenarios where evaluators do not have the luxury of comparing performance to a single right answer. This framework has been used to evaluate human-machine creative collaborations across story and music generation, interactive block building, and exploration of molecular mechanisms in cancer. These activities are fundamentally different from the more constrained tasks performed by most contemporary personal assistants as they are generally open-ended, with no single correct solution, and often no obvious completion criteria. We identified the Key Properties that must be exhibited by successful systems. From there we identified "Hallmarks" of success -- capabilities and features that evaluators can observe that would be indicative of progress toward achieving a Key Property. In addition to being a framework for assessment, the Key Properties and Hallmarks are intended to serve as goals in guiding research direction.