2018/10/21 by Siwei Fu, Fu, Siwei, Anbang Xu +7 · 1 citation
Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #Artificial intelligence #Categorization #Computer science #Crowdsourcing #Database #FOS: Computer and information sciences #Machine learning #Mobile Crowdsensing and Crowdsourcing #Process (computing) #Programming language #Software engineering #Workflow #World Wide Web #cs.AI
paper · pdf · doi:10.48550/arxiv.1810.09030
published in arXiv (Cornell University), 1-8 (Cornell University) · a 10-page full paper
arxiv created 2018/10/21 · openalex publication_date 2018/10/21 · arxiv updated 2018/10/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Artificial Intelligence (AI) has burrowed into our lives in various aspects; however, without appropriate testing, deployed AI systems are often being criticized to fail in critical and embarrassing cases. Existing testing approaches mainly depend on fixed and pre-defined datasets, providing a limited testing coverage. In this paper, we propose the concept of proactive testing to dynamically generate testing data and evaluate the performance of AI systems. We further introduce Challenge.AI, a new crowd system that features the integration of crowdsourcing and machine learning techniques in the process of error generation, error validation, error categorization, and error analysis. We present experiences and insights into a participatory design with AI developers. The evaluation shows that the crowd workflow is more effective with the help of machine learning techniques. AI developers found that our system can help them discover unknown errors made by the AI models, and engage in the process of proactive testing.