Which *BERT? A Survey Organizing Contextualized Encoders

2020/10/02 by Patrick Xia, Shijie Wu, Xia, Patrick +3 · 1 citation
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2010.00854

Abstract

Pretrained contextualized text encoders are now a staple of the NLP community. We present a survey on language representation learning with the aim of consolidating a series of shared lessons learned across a variety of recent efforts. While significant advancements continue at a rapid pace, we find that enough has now been discovered, in different directions, that we can begin to organize advances according to common themes. Through this organization, we highlight important considerations when interpreting recent contributions and choosing which model to use.

Citations

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Insertion Transformer: Flexible Sequence Generation via Insertion\n Operations
• ERNIE: Enhanced Representation through Knowledge Integration
• Predicting Parameters in Deep Learning
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• LLM-Supported Manufacturing Mapping Generation
• Reformer: The Efficient Transformer
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Scaling Laws for Neural Language Models
• Multilingual Denoising Pre-training for Neural Machine Translation

Cited by

• Polarity Detection of Sustainable Detection Goals in News Text

Related

• What the [MASK]? Making Sense of Language-Specific BERT Models
• A Primer in BERTology: What we know about how BERT works
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• Latent Universal Task-Specific BERT
• How to Fine-Tune BERT for Text Classification?