Modeling Protein Using Large-scale Pretrain Language Model

2021/08/17 by Yijia Xiao, Xiao, Yijia, Jiezhong Qiu +8 · 5 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Biomolecules (q-bio.BM) #Computation and Language (cs.CL) #FOS: Biological sciences #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Bioinformatics #Protein Structure and Dynamics #RNA and protein synthesis mechanisms #cs.CL #cs.LG #q-bio.BM

paper · pdf · doi:10.48550/arxiv.2108.07435

Accepted paper in Pretrain@KDD 2021 (The International Workshop on Pretraining: Algorithms, Architectures, and Applications)

openalex publication_date 2021/08/17 · arxiv created 2021/12/07 · arxiv updated 2021/12/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Protein is linked to almost every life process. Therefore, analyzing the biological structure and property of protein sequences is critical to the exploration of life, as well as disease detection and drug discovery. Traditional protein analysis methods tend to be labor-intensive and time-consuming. The emergence of deep learning models makes modeling data patterns in large quantities of data possible. Interdisciplinary researchers have begun to leverage deep learning methods to model large biological datasets, e.g. using long short-term memory and convolutional neural network for protein sequence classification. After millions of years of evolution, evolutionary information is encoded in protein sequences. Inspired by the similarity between natural language and protein sequences, we use large-scale language models to model evolutionary-scale protein sequences, encoding protein biology information in representation. Significant improvements are observed in both token-level and sequence-level tasks, demonstrating that our large-scale model can accurately capture evolution information from pretraining on evolutionary-scale individual sequences. Our code and model are available at https://github.com/THUDM/ProteinLM.

Citations

• Efficient Estimation of Word Representations in Vector Space
• Protein Structure Prediction by Protein Alignments
• Evaluating Protein Transfer Learning with TAPE
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Cited by

• Generalizable and scalable protein stability prediction with rewired protein generative models
• Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Model
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• GLProtein: Global-and-Local Structure Aware Protein Representation Learning
• Elucidating the Design Space of Multimodal Protein Language Models

Related

• Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
• Design Proteins Using Large Language Models: Enhancements and Comparative Analyses
• A Fine-tuning Dataset and Benchmark for Large Language Models for Protein Understanding
• ProteinBERT: a universal deep-learning model of protein sequence and function
• Protein Large Language Models: A Comprehensive Survey