ProteinCLIP: enhancing protein language models with natural language

2024/05/17 by Kevin Wu, Howard F. Chang, James Zou · 1 voice · 2 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Biomedical Text Mining and Ontologies #Machine Learning in Bioinformatics #Topic Modeling

paper · pdf · doi:10.1101/2024.05.14.594226

openalex publication_date 2024/05/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/14

Abstract

Abstract Language models have enabled a new era of biological sequence modeling. However, extracting meaningful sequence-level embeddings from these models remains challenging. In this work, we introduce ProteinCLIP, which applies contrastive learning between a protein’s amino acid sequence and curated text describing its function. ProteinCLIP thus learns to take a pre-trained protein language model’s sequence embedding and refines it produce a function-centric embedding. We show that this embedding space yields sequence representations that enable state-of-the-art performance across a variety of important yet challenging tasks in the study of proteins – from predicting protein protein interactions to accurately detecting homologous proteins despite low sequence similarity. More broadly, ProteinCLIP demonstrates the effectiveness of multi-modal learning in biological contexts, and how such strategies can help isolate key signals from large models and further improve their utility.

Citations

• SciPy 1.0: fundamental algorithms for scientific computing in Python
• A Simple Framework for Contrastive Learning of Visual Representations
• A SARS-CoV-2 protein interaction map reveals targets for drug repurposing
• Array programming with NumPy
• Array programming with NumPy
• DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome
• Learning Transferable Visual Models From Natural Language Supervision
• Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
• ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
• Highly accurate protein structure prediction with AlphaFold
• Effective gene expression prediction from sequence by integrating long-range interactions
• High-Resolution Image Synthesis with Latent Diffusion Models
• Evolutionary-scale prediction of atomic-level protein structure with a language model
• Transfer learning enables predictions in network biology
• De novo design of protein structure and function with RFdiffusion
• Cracking the black box of deep sequence-based protein–protein interaction prediction

Cited by

• Deep learning models for unbiased sequence-based PPI prediction plateau at an accuracy of 0.65
• Protein as a Second Language for LLMs

Discussions

• This last point is corroborated by work by Wu et al earlier this year, who found that embeddings derived from larger PLMs are more accurate when fine-tuned on structural-similarity searches. doi.org/1 [bsky, 4 points, 1 comments]

Related

• Codon language embeddings provide strong signals for protein engineering
• Prot2Text-V2: Protein Function Prediction with Multimodal Contrastive Alignment
• ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Texts
• A Fine-tuning Dataset and Benchmark for Large Language Models for Protein Understanding
• Modeling Protein Using Large-scale Pretrain Language Model