2021/05/01 by Haifan Gong, Gong, Haifan, Guanqi Chen +7 · 1 citation
Computer Science · #Advanced Image and Video Retrieval Techniques #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimedia (cs.MM) #Multimodal Machine Learning Applications
paper · pdf · doi:10.48550/arxiv.2105.00136
openalex publication_date 2021/05/01 · openalex created_date 2021/05/10 · openalex updated_date 2026/07/31
Due to the severe lack of labeled data, existing methods of medical visual question answering usually rely on transfer learning to obtain effective image feature representation and use cross-modal fusion of visual and linguistic features to achieve question-related answer prediction. These two phases are performed independently and without considering the compatibility and applicability of the pre-trained features for cross-modal fusion. Thus, we reformulate image feature pre-training as a multi-task learning paradigm and witness its extraordinary superiority, forcing it to take into account the applicability of features for the specific image comprehension task. Furthermore, we introduce a cross-modal self-attention~(CMSA) module to selectively capture the long-range contextual relevance for more effective fusion of visual and linguistic features. Experimental results demonstrate that the proposed method outperforms existing state-of-the-art methods. Our code and models are available at https://github.com/haifangong/CMSA-MTPT-4-MedicalVQA.