2019/07/29 by Damien Teney, Peng Wang, Teney, Damien +9 · 2 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image Retrieval and Classification Techniques #Multimodal Machine Learning Applications
paper · pdf · doi:10.48550/arxiv.1907.12271
openalex publication_date 2019/07/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
One of the primary challenges faced by deep learning is the degree to which\ncurrent methods exploit superficial statistics and dataset bias, rather than\nlearning to generalise over the specific representations they have experienced.\nThis is a critical concern because generalisation enables robust reasoning over\nunseen data, whereas leveraging superficial statistics is fragile to even small\nchanges in data distribution. To illuminate the issue and drive progress\ntowards a solution, we propose a test that explicitly evaluates abstract\nreasoning over visual data. We introduce a large-scale benchmark of visual\nquestions that involve operations fundamental to many high-level vision tasks,\nsuch as comparisons of counts and logical operations on complex visual\nproperties. The benchmark directly measures a method's ability to infer\nhigh-level relationships and to generalise them over image-based concepts. It\nincludes multiple training/test splits that require controlled levels of\ngeneralization. We evaluate a range of deep learning architectures, and find\nthat existing models, including those popular for vision-and-language tasks,\nare unable to solve seemingly-simple instances. Models using relational\nnetworks fare better but leave substantial room for improvement.\n