2016/12/14 by Dong Huk Park, Lisa Anne Hendricks, Park, Dong Huk +9 · 6 citations
Computer Science · #Multimodal Machine Learning Applications #Visual Attention and Saliency Detection #Domain Adaptation and Few-Shot Learning
paper · pdf · doi:10.48550/arxiv.1612.04757
Deep models are the defacto standard in visual decision models due to their\nimpressive performance on a wide array of visual tasks. However, they are\nfrequently seen as opaque and are unable to explain their decisions. In\ncontrast, humans can justify their decisions with natural language and point to\nthe evidence in the visual world which led to their decisions. We postulate\nthat deep models can do this as well and propose our Pointing and Justification\n(PJ-X) model which can justify its decision with a sentence and point to the\nevidence by introspecting its decision and explanation process using an\nattention mechanism. Unfortunately there is no dataset available with reference\nexplanations for visual decision making. We thus collect two datasets in two\ndomains where it is interesting and challenging to explain decisions. First, we\nextend the visual question answering task to not only provide an answer but\nalso a natural language explanation for the answer. Second, we focus on\nexplaining human activities which is traditionally more challenging than object\nclassification. We extensively evaluate our PJ-X model, both on the\njustification and pointing tasks, by comparing it to prior models and ablations\nusing both automatic and human evaluations.\n