Multi-Task Learning in Natural Language Processing: An Overview

2021/09/19 by Shijie Chen, Yu Zhang, Chen, Shijie +3 · 9 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2109.09138

openalex publication_date 2021/09/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep learning approaches have achieved great success in the field of Natural Language Processing (NLP). However, directly training deep neural models often suffer from overfitting and data scarcity problems that are pervasive in NLP tasks. In recent years, Multi-Task Learning (MTL), which can leverage useful information of related tasks to achieve simultaneous performance improvement on these tasks, has been used to handle these problems. In this paper, we give an overview of the use of MTL in NLP tasks. We first review MTL architectures used in NLP tasks and categorize them into four classes, including parallel architecture, hierarchical architecture, modular architecture, and generative adversarial architecture. Then we present optimization techniques on loss construction, gradient regularization, data sampling, and task scheduling to properly train a multi-task model. After presenting applications of MTL in a variety of NLP tasks, we introduce some benchmark datasets. Finally, we make a conclusion and discuss several possible research directions in this field.

Citations

• All-in-one: Multi-task Learning for Rumour Verification
• Meta Multi-Task Learning for Sequence Modeling
• A Survey on Multi-Task Learning

Cited by

• Semimage: HSV-Based Semantic Image Encoding for Disentangled Text Representation
• POLIS-Bench: Towards Multi-Dimensional Evaluation of LLMs for Bilingual Policy Tasks in Governmental Scenarios
• NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective
• Fairness Metric Design Exploration in Multi-Domain Moral Sentiment Classification using Transformer-Based Models
• STEPER: Step-wise Knowledge Distillation for Enhancing Reasoning Ability in Multi-Step Retrieval-Augmented Language Models
• DARD: Dice Adversarial Robustness Distillation against Adversarial Attacks
• Small transformer architectures for task switching
• Efficient Inter-Task Attention for Multitask Transformer Models
• Basic Reading Distillation

Related

• Empirical Evaluation of Multi-task Learning in Deep Neural Networks for Natural Language Processing
• A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods
• Identifying beneficial task relations for multi-task learning in deep neural networks
• Multi-Task Learning with Deep Neural Networks: A Survey
• An Overview of Multi-Task Learning in Deep Neural Networks