2021/05/31 by Giang V. Nguyen, Daeyoung Kim, Nguyen, Giang +4 · 5 citations
Computer Science · Medicine · #Adversarial Robustness in Machine Learning #Domain Adaptation and Few-Shot Learning #Artificial Intelligence in Healthcare and Education
paper · pdf · doi:10.48550/arxiv.2105.14944
Explaining the decisions of an Artificial Intelligence (AI) model is\nincreasingly critical in many real-world, high-stake applications. Hundreds of\npapers have either proposed new feature attribution methods, discussed or\nharnessed these tools in their work. However, despite humans being the target\nend-users, most attribution methods were only evaluated on proxy\nautomatic-evaluation metrics (Zhang et al. 2018; Zhou et al. 2016; Petsiuk et\nal. 2018). In this paper, we conduct the first user study to measure\nattribution map effectiveness in assisting humans in ImageNet classification\nand Stanford Dogs fine-grained classification, and when an image is natural or\nadversarial (i.e., contains adversarial perturbations). Overall, feature\nattribution is surprisingly not more effective than showing humans nearest\ntraining-set examples. On a harder task of fine-grained dog categorization,\npresenting attribution maps to humans does not help, but instead hurts the\nperformance of human-AI teams compared to AI alone. Importantly, we found\nautomatic attribution-map evaluation measures to correlate poorly with the\nactual human-AI team performance. Our findings encourage the community to\nrigorously test their methods on the downstream human-in-the-loop applications\nand to rethink the existing evaluation metrics.\n