2018/11/14 by Victor Sanh, Thomas Wolf, Sanh, Victor +3
Computer Science · Decision Sciences · #Computation and Language (cs.CL) #Data Quality and Management #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.1811.06031
openalex publication_date 2018/11/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Much effort has been devoted to evaluate whether multi-task learning can be\nleveraged to learn rich representations that can be used in various Natural\nLanguage Processing (NLP) down-stream applications. However, there is still a\nlack of understanding of the settings in which multi-task learning has a\nsignificant effect. In this work, we introduce a hierarchical model trained in\na multi-task learning setup on a set of carefully selected semantic tasks. The\nmodel is trained in a hierarchical fashion to introduce an inductive bias by\nsupervising a set of low level tasks at the bottom layers of the model and more\ncomplex tasks at the top layers of the model. This model achieves\nstate-of-the-art results on a number of tasks, namely Named Entity Recognition,\nEntity Mention Detection and Relation Extraction without hand-engineered\nfeatures or external NLP tools like syntactic parsers. The hierarchical\ntraining supervision induces a set of shared semantic representations at lower\nlayers of the model. We show that as we move from the bottom to the top layers\nof the model, the hidden states of the layers tend to represent more complex\nsemantic information.\n