Domain Pre-training Impact on Representations

2025/05/30 by Gonzalez-Gutierrez, Cesar, Quattoni, Ariadna
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2505.24455

Abstract

This empirical study analyzes the effects of the pre-training corpus on the quality of learned transformer representations. We focus on the representation quality induced solely through pre-training. Our experiments show that pre-training on a small, specialized corpus can yield effective representations, and that the success of combining a generic and a specialized corpus depends on the distributional similarity between the target task and the specialized corpus.

Citations

• Probing the Emergence of Cross-lingual Alignment during LLM Training
• On the Generalization Ability of Unsupervised Pretraining
• Analyzing Text Representations by Measuring Task Alignment
• On the Provable Advantage of Unsupervised Pretraining
• Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study
• Downstream Datasets Make Surprisingly Good Pretraining Corpora
• Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models
• Probing Across Time: What Does RoBERTa Know and When?
• DirectProbe: Studying Representations without Classifiers
• When Do You Need Billions of Words of Pretraining Data?
• HateBERT: Retraining BERT for Abusive Language Detection in English
• Pretrained Language Model Embryology: The Birth of ALBERT
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• On the Theory of Transfer Learning: The Importance of Task Diversity
• Array Programming with NumPy
• BERTweet: A pre-trained language model for English Tweets
• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
• Unsupervised Domain Clusters in Pretrained Language Models
• A Primer in BERTology: What we know about how BERT works
• Few-Shot Learning via Learning the Representation, Provably
• How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis
• Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification
• Linguistic Knowledge and Transferability of Contextual Representations
• What you can cram into a single vector: Probing sentence embeddings for linguistic properties
• Deep contextualized word representations
• Ex Machina: Personal Attacks Seen at Scale
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• Character-level Convolutional Networks for Text Classification
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• On Initializing Transformers with Pre-trained Embeddings
• The Effects of In-domain Corpus Size on pre-training BERT
• Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities
• Don't Sweep your Learning Rate under the Rug: A Closer Look at Cross-modal Transfer of Pretrained Transformers
• Pre-Training Transformers for Domain Adaptation