Advancing bioinformatics with language models: components, applications, and perspectives

2024/01/08 by Jiajia Liu, Liu, Jiajia, Mengyuan Yang +12 · 1 voice · 1 citation
Biochemistry, Genetics and Molecular Biology · Computer Science · #Single-cell and spatial transcriptomics #Machine Learning in Bioinformatics #Topic Modeling

paper · pdf · doi:10.1093/bib/bbag367

Abstract

Large language models (LLMs) are deep learning-based artificial intelligence models that have achieved remarkable success in natural language processing. Typically composed of neural networks with billions of parameters, they are trained on massive unlabeled datasets using self-supervised or semi-supervised learning. Beyond language, LLMs hold immense potential for addressing complex bioinformatics challenges. This review provides a comprehensive overview of transformer-based model applications in genomics, transcriptomics, proteomics, drug discovery, and single-cell analysis. We discuss critical components, including tokenization strategies for diverse biological data, transformer architectures, attention mechanisms, and pretraining approaches. We also survey currently available foundation models and their downstream applications across bioinformatics domains. Finally, we highlight major challenges that remain insufficiently addressed in prior reviews and outline future perspectives and design principles for next-generation biological language models, offering practical guidance for both users and developers.

Citations

• A DNA language model based on multispecies alignment predicts the effects of genome-wide variants
• Genomic language models: opportunities and challenges
• Transfer learning enables predictions in network biology
• LoRA: Low-Rank Adaptation of Large Language Models
• Energy and Policy Considerations for Deep Learning in NLP
• Attention Is All You Need
• Distilling the Knowledge in a Neural Network
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Large Language Models: A Paradigm Shift for Dementia Diagnosis and Care

Discussions

• “Large language models in bioinformatics: applications and perspectives” Summarizes different uses and architectures, particularly highlighting difference between BERT and GPT-type models arxiv.org/ [bsky, 3 points, 0 comments]

Related

• Large language models and their applications in bioinformatics
• Large Language Models in Bioinformatics: A Survey
• Large Language Models for Bioinformatics
• Scientific Large Language Models: A Survey on Biological & Chemical Domains
• An Evaluation of Large Language Models in Bioinformatics Research