Do Text-to-Text Multi-Task Learners Suffer from Task Conflict?

2022/12/13 by David Mueller, Nicholas Andrews, Mueller, David +3 · 1 citation
Computer Science · #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning and Data Classification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2212.06645

openalex publication_date 2022/12/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Traditional multi-task learning architectures train a single model across multiple tasks through a shared encoder followed by task-specific decoders. Learning these models often requires specialized training algorithms that address task-conflict in the shared parameter updates, which otherwise can lead to negative transfer. A new type of multi-task learning within NLP homogenizes multi-task architectures as a shared encoder and language model decoder, which does surprisingly well across a range of diverse tasks. Does this new architecture suffer from task-conflicts that require specialized training algorithms? We study how certain factors in the shift towards text-to-text models affects multi-task conflict and negative transfer, finding that both directional conflict and transfer are surprisingly constant across architectures.

Cited by

• IL-PCSR: Legal Corpus for Prior Case and Statute Retrieval

Related

• Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models
• A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
• Making Small Language Models Better Multi-task Learners with Mixture-of-Task-Adapters
• Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data
• Optimizing Multi-Task Learning for Enhanced Performance in Large Language Models