Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability

2021/03/12 by Kao, Wei-Tsung, Lee, Hung-Yi
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2103.07162

Abstract

This paper investigates whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications. To verify pre-trained models' transferability, we test the pre-trained models on text classification tasks with meanings of tokens mismatches, and real-world non-text token sequence classification data, including amino acid, DNA, and music. We find that even on non-text data, the models pre-trained on text converge faster, perform better than the randomly initialized models, and only slightly worse than the models using task-specific knowledge. We also find that the representations of the text and non-text pre-trained models share non-trivial similarities.

Related

• How to Fine-Tune BERT for Text Classification?
• On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets
• Lessons Learned from Applying off-the-shelf BERT: There is no Silver Bullet
• Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models
• A Comprehensive Comparison of Pre-training Language Models