Transformers and genome language models

2025/03/13 by Micaela Elisa Consens, Micaela E. Consens, Cameron Dufault +9 · 1 voice · 4 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Evolutionary Algorithms and Applications #Genomics and Phylogenetic Studies #Machine Learning in Bioinformatics

paper · doi:10.1038/s42256-025-01007-9

crossref issued 2025/03/13 · crossref published 2025/03/13 · crossref published-online 2025/03/13 · openalex publication_date 2025/03/13 · crossref created 2025/03/13 · crossref deposited 2025/03/27 · openalex created_date 2025/10/10 · crossref indexed 2026/07/29 · openalex updated_date 2026/07/31

Citations

• Finding Structure in Time
• Deep learning: new computational modelling techniques for genomics
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• On a Method to Measure Supervised Multiclass Model’s Interpretability: Application to Degradation Diagnosis (Short Paper)
• Scaling Laws for Neural Language Models
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Expanded encyclopaedias of DNA elements in the human and mouse genomes
• DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome
• Are Pre-trained Convolutions Better than Pre-trained Transformers?
• LoRA: Low-Rank Adaptation of Large Language Models
• Highly accurate protein structure prediction with AlphaFold
• On the Opportunities and Risks of Foundation Models
• Effective gene expression prediction from sequence by integrating long-range interactions
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics
• Mega: Moving Average Equipped Gated Attention
• Token Merging: Your ViT But Faster
• Hyena Hierarchy: Towards Larger Convolutional Language Models
• Transfer learning enables predictions in network biology
• DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome
• HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution
• Retentive Network: A Successor to Transformer for Large Language Models
• Exploiting Generative AI to Scale up Intelligent Tutoring Systems
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
• Pedoman Peri-operatif Renal Assessment
• Sequence modeling and design from molecular to genome scale with Evo
• A DNA language model based on multispecies alignment predicts the effects of genome-wide variants

Cited by

• Interpretation, extrapolation and perturbation of single cells
• From Likelihood to Fitness: Improving Variant Effect Prediction in Protein and Genome Language Models
• Deep learning applications advance plant genomics research
• Mut-BPE: A Modified BPE Strategy Improves Variant Effect Prediction

Discussions

• “Transformers and genome language models” from Micaela E. Consens, Cameron Dufault, Michael Wainberg, Duncan Forster, Mehran Karimzadeh, Hani Goodarzi, Fabian J. Theis, Alan Moses & Bo Wang in Nature [bsky, 1 points, 0 comments]