ProGen: Language Modeling for Protein Generation

2020/03/08 by Ali Madani, Madani, Ali, Bryan McCann +13 · 14 citations
Biochemistry, Genetics and Molecular Biology · #Biomolecules (q-bio.BM) #FOS: Biological sciences #FOS: Computer and information sciences #Genomics and Phylogenetic Studies #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Bioinformatics #RNA and protein synthesis mechanisms

paper · pdf · doi:10.48550/arxiv.2004.03497

openalex publication_date 2020/03/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Generative modeling for protein engineering is key to solving fundamental problems in synthetic biology, medicine, and material science. We pose protein engineering as an unsupervised sequence generation problem in order to leverage the exponentially growing set of proteins that lack costly, structural annotations. We train a 1.2B-parameter language model, ProGen, on ~280M protein sequences conditioned on taxonomic and keyword tags such as molecular function and cellular component. This provides ProGen with an unprecedented range of evolutionary sequence diversity and allows it to generate with fine-grained control as demonstrated by metrics based on primary sequence similarity, secondary structure accuracy, and conformational energy.

Citations

• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Cited by

• Self Distillation Fine-Tuning of Protein Language Models Improves Versatility in Protein Design
• Full-Atom Peptide Design via Riemannian-Euclidean Bayesian Flow Networks
• How to make the most of your masked language model for protein engineering
• FrustrAI-Seq: Scaling Local Energetic Frustration to the Protein Sequence Space
• Deep Extrapolation for Attribute-Enhanced Generation
• Generative AI for Biosciences: Emerging Threats and Roadmap to Biosecurity
• Physicochemically Informed Dual-Conditioned Generative Model of T-Cell Receptor Variable Regions for Cellular Therapy
• AdaLead: A simple and robust adaptive greedy search algorithm for sequence design
• From Prediction to Simulation: AlphaFold 3 as a Differentiable Framework for Structural Biology
• Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning
• Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Model
• Learning the language of protein-protein interactions
• Pi-SAGE: Permutation-invariant surface-aware graph encoder for binding affinity prediction
• A Comprehensive Review of Transformer-based language models for Protein Sequence Analysis and Design

Related

• ProGen2: Exploring the Boundaries of Protein Language Models
• ProtGPT2 is a deep unsupervised language model for protein design
• ProLLaMA: A Protein Large Language Model for Multi-Task Protein Language Processing
• Exploring the Protein Sequence Space with Global Generative Models
• De Novo Functional Protein Sequence Generation: Overcoming Data Scarcity through Regeneration and Large Models