Meta-learning For Vision-and-language Cross-lingual Transfer

2023/05/24 by Hu, Hanxu, Keller, Frank · 2 citations
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2305.14843

Abstract

Current pre-trained vison-language models (PVLMs) achieve excellent performance on a range of multi-modal datasets. Recent work has aimed at building multilingual models, and a range of novel multilingual multi-modal datasets have been proposed. Current PVLMs typically perform poorly on these datasets when used for multi-modal zero-shot or few-shot cross-lingual transfer, especially for low-resource languages. To alleviate this problem, we propose a novel meta-learning fine-tuning framework. Our framework makes current PVLMs rapidly adaptive to new languages in vision-language scenarios by designing MAML in a cross-lingual multi-modal manner. Experiments show that our method boosts the performance of current state-of-the-art PVLMs in both zero-shot and few-shot cross-lingual transfer on a range of vision-language understanding tasks and datasets (XVNLI, xGQA, MaRVL, xFlicker&Co)

Cited by

• Privacy Preserving Ordinal-Meta Learning with VLMs for Fine-Grained Fruit Quality Prediction
• Multilingual Vision-Language Models, A Survey

Related

• Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment
• Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training
• MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching
• CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer
• UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training