2022/05/14 by Lovisa Hagström, Hagström, Lovisa, Richard Johansson +1
Computer Science · #Artificial intelligence #Commonsense knowledge #Commonsense reasoning #Computation and Language (cs.CL) #Computer science #FOS: Computer and information sciences #Focus (optics) #Knowledge extraction #Language model #Language understanding #Modality (human–computer interaction) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Natural language processing #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2205.07065
openalex publication_date 2022/05/14 · openalex created_date 2022/05/22 · openalex updated_date 2026/08/05
There are limitations in learning language from text alone. Therefore, recent focus has been on developing multimodal models. However, few benchmarks exist that can measure what language models learn about language from multimodal training. We hypothesize that training on a visual modality should improve on the visual commonsense knowledge in language models. Therefore, we introduce two evaluation tasks for measuring visual commonsense knowledge in language models and use them to evaluate different multimodal models and unimodal baselines. Primarily, we find that the visual commonsense knowledge is not significantly different between the multimodal models and unimodal baseline models trained on visual text data.