Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Prompts

2023/05/11 by Zhang, Zhaoyang, Shen, Yantao, Shi, Kunyu +7 · 1 citation
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2305.07019

Abstract

We present a vision-language model whose parameters are jointly trained on all tasks and fully shared among multiple heterogeneous tasks which may interfere with each other, resulting in a single model which we named Musketeer. The integration of knowledge across heterogeneous tasks is enabled by a novel feature called Task Explanation Prompt (TEP). With rich and structured information such as task input/output format, TEP reduces interference among tasks, allowing the model to focus on their shared structure. With a single model, Musketeer achieves results comparable to or better than strong baselines trained on single tasks, almost uniformly across multiple tasks.

Cited by

• Task-Specific Dual-Model Framework for Comprehensive Traffic Safety Video Description and Analysis

Related

• A Retrospect to Multi-prompt Learning across Vision and Language
• Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants
• Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
• 12-in-1: Multi-Task Vision and Language Representation Learning
• Multitask Vision-Language Prompt Tuning