Multi-Task Learning for Sequence Tagging: An Empirical Study

2018/08/13 by Soravit Changpinyo, Hexiang Hu, Changpinyo, Soravit +3 · 4 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1808.04151

openalex publication_date 2018/08/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We study three general multi-task learning (MTL) approaches on 11 sequence tagging tasks. Our extensive empirical results show that in about 50% of the cases, jointly learning all 11 tasks improves upon either independent or pairwise learning of the tasks. We also show that pairwise MTL can inform us what tasks can benefit others or what tasks can be benefited if they are learned jointly. In particular, we identify tasks that can always benefit others as well as tasks that can always be harmed by others. Interestingly, one of our MTL approaches yields embeddings of the tasks that reveal the natural clustering of semantic and syntactic tasks. Our inquiries have opened the doors to further utilization of MTL in NLP.

Citations

• On the difficulty of training Recurrent Neural Networks
• Sequence to Sequence Learning with Neural Networks
• An Overview of Multi-Task Learning in Deep Neural Networks

Cited by

• An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining
• Deconstructing and reconstructing word embedding algorithms
• A Framework For Identifying Group Behavior Of Wild Animals
• Learning Functions to Study the Benefit of Multitask Learning

Related

• When is multitask learning effective? Semantic sequence prediction under varying data conditions
• What can we learn from Semantic Tagging?
• Multitask Learning for Sequence Labeling Tasks
• Multi-task Sequence to Sequence Learning
• Learning Functions to Study the Benefit of Multitask Learning