Protein Language Models: Is Scaling Necessary?

2024/09/23 by Quentin Fournier, Robert M. Vernon, Almer M. van der Sloot +3 · 1 voice · 4 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Machine Learning in Bioinformatics #Natural Language Processing Techniques #Topic Modeling

paper · doi:10.1101/2024.09.23.614603

openalex publication_date 2024/09/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/14

Abstract

Public protein sequence databases contain samples from the fitness landscape explored by nature. Protein language models (pLMs) pre-trained on these sequences aim to capture this landscape for tasks like property prediction and protein design. Following the same trend as in natural language processing, pLMs have continuously been scaled up. However, the premise that scale leads to better performance assumes that source databases provide an accurate representation of the underlying fitness landscape, which is likely false. By developing an efficient codebase, designing a modern architecture, and addressing data quality concerns such as sample bias, we introduce AMPLIFY, a best-in-class pLM that is orders of magnitude less expensive to train and deploy than previous models. Furthermore, to support the scientific community and democratize the training of pLMs, we have open-sourced AMPLIFY’s pre-training codebase, data, and model checkpoints.

Citations

• Dictionary of protein secondary structure: Pattern recognition of hydrogen‐bonded and geometrical features
• Improved protein structure prediction using potentials from deep learning
• Scaling Laws for Neural Language Models
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
• Highly accurate protein structure prediction with AlphaFold
• Learning inverse folding from millions of predicted structures
• Training Compute-Optimal Large Language Models
• Robust deep learning–based protein sequence design using ProteinMPNN
• Long non-coding RNAs: definitions, functions, challenges and recommendations
• Evolutionary-scale prediction of atomic-level protein structure with a language model
• The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
• De novo design of protein structure and function with RFdiffusion
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• ProGen2: Exploring the boundaries of protein language models
• Accurate structure prediction of biomolecular interactions with AlphaFold 3
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Cited by

• Leveraging learned representations and multitask learning for lysine methylation site discovery
• Scaling and Data Saturation in Protein Language Models
• Sequence-based protein-protein interaction prediction and its applications in drug discovery
• Structure-Aligned Protein Language Model

Discussions

• Similar findings in this one doi.org/10.1101/2024.... There are many more influences at play than mere scaling of parameters... [bsky, 3 points, 0 comments]

Related

• Scaling and Data Saturation in Protein Language Models
• Are Protein Language Models Compute Optimal?
• Training Compute-Optimal Protein Language Models
• Training Compute-Optimal Protein Language Models
• A Comprehensive Review of Protein Language Models