A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

2016/11/05 by Kazuma Hashimoto, Caiming Xiong, Hashimoto, Kazuma +5 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.1611.01587

Accepted as a full paper at the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017)

arxiv created 2017/07/24 · arxiv updated 2017/07/25

Abstract

Transfer and multi-task learning have traditionally focused on either a single source-target pair or very few, similar tasks. Ideally, the linguistic levels of morphology, syntax and semantics would benefit each other by being trained in a single model. We introduce a joint many-task model together with a strategy for successively growing its depth to solve increasingly complex tasks. Higher layers include shortcut connections to lower-level task predictions to reflect linguistic hierarchies. We use a simple regularization term to allow for optimizing all model weights to improve one task's loss without exhibiting catastrophic interference of the other tasks. Our single end-to-end model obtains state-of-the-art or competitive results on five different tasks from tagging, parsing, relatedness, and entailment tasks.

Cited by

• Translation via Annotation: A Computational Study of Translating Classical Chinese into Japanese
• Measuring Fine-Grained Relatedness in Multitask Learning via Data Attribution

Related

• Multi-Task Learning in Natural Language Processing: An Overview
• Identifying beneficial task relations for multi-task learning in deep neural networks
• TaskWeb: Selecting Better Source Tasks for Multi-task NLP
• Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data
• A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods