One protein is all you need

2024/11/04 by Anton Bushuiev, Bushuiev, Anton, Roman Bushuiev +24 · 3 voices · 2 citations
Biochemistry, Genetics and Molecular Biology · Chemistry · Computer Science · #Advanced Proteomics Techniques and Applications #Artificial intelligence #Biology #Computer science #Ecology #Evolutionary biology #Fitness function #Function (biology) #Genetic algorithm #Machine learning #Physics #Protein structure #Protein structure prediction #Test (biology) #Training (meteorology) #cs.LG #q-bio.BM

paper · pdf · doi:10.48550/arxiv.2411.02109

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/11/04 · openalex created_date 2024/11/15 · openalex updated_date 2026/08/03

Abstract

Generalization beyond training data remains a central challenge in machine learning for biology. A common way to enhance generalization is self-supervised pre-training on large datasets. However, aiming to perform well on all possible proteins can limit a model's capacity to excel on any specific one, whereas experimentalists typically need accurate predictions for individual proteins they study, often not covered in training data. To address this limitation, we propose a method that enables self-supervised customization of protein language models to one target protein at a time, on the fly, and without assuming any additional data. We show that our Protein Test-Time Training (ProteinTTT) method consistently enhances generalization across different models, their sizes, and datasets. ProteinTTT improves structure prediction for challenging targets, achieves new state-of-the-art results on protein fitness prediction, and enhances function prediction on two tasks. Through two challenging case studies, we also show that customization via ProteinTTT achieves more accurate antibody-antigen loop modeling and enhances 19% of structures in the Big Fantastic Virus Database, delivering improved predictions where general-purpose AlphaFold2 and ESMFold struggle.

Cited by

• Democratizing Protein Language Model Training, Sharing and Collaboration

Discussions

• We train machine learning models on millions of proteins. But when it comes to making predictions, do we need them to understand all proteins at once? Often, we need an accurate model for the specific [bsky, 69 points, 2 comments]
• This is weird and unexpected: fine-tuning the base ESM2 protein language model on a test sequence of interest can lead to improved prediction of that protein's structure, as well as improved variant e [bsky, 20 points, 3 comments]
• We have a little new paper at ICLR led by @AntonBushuiev. Test time training for proteins :) arxiv.org/abs/2411.02109 For example, you know the sequence that you want to fold, so fine-tune ESM on it a [bsky, 8 points, 0 comments]

Related

• Enhancing the efficiency of protein language models with minimal wet-lab data through few-shot learning
• ProGen2: Exploring the Boundaries of Protein Language Models
• Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval
• Metalic: Meta-Learning In-Context with Protein Language Models
• Training Compute-Optimal Protein Language Models