Deepening Hidden Representations from Pre-trained Language Models

2019/11/05 by Junjie Yang, Hai Zhao, Yang, Junjie +1 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1911.01940

openalex publication_date 2019/11/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Transformer-based pre-trained language models have proven to be effective for learning contextualized language representation. However, current approaches only take advantage of the output of the encoder's final layer when fine-tuning the downstream tasks. We argue that only taking single layer's output restricts the power of pre-trained representation. Thus we deepen the representation learned by the model by fusing the hidden representation in terms of an explicit HIdden Representation Extractor (HIRE), which automatically absorbs the complementary representation with respect to the output from the final layer. Utilizing RoBERTa as the backbone encoder, our proposed improvement over the pre-trained models is shown effective on multiple natural language understanding tasks and help our model rival with the state-of-the-art models on the GLUE benchmark.

Citations

• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Cited by

• Hierarchical Multitask Learning Approach for BERT

Related

• Generate to Understand for Representation
• Pre-trained Language Model Representations for Language Generation
• Pre-Training a Language Model Without Human Language
• Contextual Representation Learning beyond Masked Language Modeling
• Supervised Contextual Embeddings for Transfer Learning in Natural Language Processing Tasks