ProGen2: Exploring the Boundaries of Protein Language Models

2022/06/27 by Erik Nijkamp, Jeffrey A. Ruffolo, Nijkamp, Erik +7 · 24 citations
Biochemistry, Genetics and Molecular Biology · #FOS: Biological sciences #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning in Bioinformatics #Quantitative Methods (q-bio.QM) #RNA and protein synthesis mechanisms #vaccines and immunoinformatics approaches

paper · pdf · doi:10.48550/arxiv.2206.13517

openalex publication_date 2022/06/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Attention-based models trained on protein sequences have demonstrated incredible success at classification and generation tasks relevant for artificial intelligence-driven protein design. However, we lack a sufficient understanding of how very large-scale models and data play a role in effective protein model development. We introduce a suite of protein language models, named ProGen2, that are scaled up to 6.4B parameters and trained on different sequence datasets drawn from over a billion proteins from genomic, metagenomic, and immune repertoire databases. ProGen2 models show state-of-the-art performance in capturing the distribution of observed evolutionary sequences, generating novel viable sequences, and predicting protein fitness without additional finetuning. As large model sizes and raw numbers of protein sequences continue to become more widely accessible, our results suggest that a growing emphasis needs to be placed on the data distribution provided to a protein sequence model. We release the ProGen2 models and code at https://github.com/salesforce/progen.

Cited by

• Few-shot Protein Fitness Prediction via In-context Learning and Test-time Training
• Mechanistic Interpretability of Antibody Language Models Using SAEs
• Swarms of Large Language Model Agents for Protein Sequence Design with Experimental Validation
• A Diffusion Model to Shrink Proteins While Maintaining Their Function
• How far can you go? Extrapolating values of catalytic activity from known protein landscapes in natural and directed evolution
• Artificial intelligence in bioinformatics: a survey
• LDBT instead of DBTL: combining machine learning and rapid cell-free testing
• Quest for Orthologs in the era of Data Deluge and AI: Challenges and Innovations in Orthology Prediction and Data Integration
• Protein as a Second Language for LLMs
• Generative AI for Biosciences: Emerging Threats and Roadmap to Biosecurity
• Evolutionary Profiles for Protein Fitness Prediction
• SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation
• Cross-species gene redesign leveraging ortholog information and generative modeling
• Active learning-guided optimization of cell-free biosensors for lead testing in drinking water
• Planner Aware Path Learning in Diffusion Language Models Training
• Codon language embeddings provide strong signals for protein engineering
• SpecMER: Fast Protein Generation with K-mer Guided Speculative Decoding
• ShortListing Model: A Streamlined SimplexDiffusion for Discrete Variable Generation
• A deep reinforcement learning platform for antibiotic discovery
• Genome-wide prediction of disease variant effects with a deep protein language model
• Scaling and Data Saturation in Protein Language Models
• A Comprehensive Review of Transformer-based language models for Protein Sequence Analysis and Design
• Steering Protein Language Models
• DisProtEdit: Exploring Disentangled Representations for Multi-Attribute Protein Editing

Related

• ProGen: Language Modeling for Protein Generation
• Training Compute-Optimal Protein Language Models
• Training Compute-Optimal Protein Language Models
• Protein Language Models: Is Scaling Necessary?
• Design Proteins Using Large Language Models: Enhancements and Comparative Analyses