Pangenome-Informed Language Models for Synthetic Genome Sequence Generation

2024/09/20 by P.-S. Huang, François Charton, Jan-Niklas Schmelzle +4 · 1 voice
Computer Science · Biochemistry, Genetics and Molecular Biology · #Privacy-Preserving Technologies in Data #Cancer Genomics and Diagnostics #Epigenetics and DNA Methylation

paper · doi:10.1101/2024.09.18.612131

openalex created_date 2024/09/20 · openalex publication_date 2024/09/20 · openalex updated_date 2026/07/14

Abstract

Language Models (LM) have been extensively utilized for learning DNA sequence patterns and generating synthetic sequences. In this paper, we present a novel approach for the generation of synthetic DNA data using pangenomes in combination with LM. We introduce three innovative pangenome-based tokenization schemes that enhance DNA sequence generation. Our experimental results demonstrate the superiority of pangenome-based tokenization over classical methods in generating high-utility synthetic DNA sequences, highlighting significant improvements in training efficiency and sequence quality.

Citations

• The Cancer Genome Atlas Pan-Cancer analysis project
• Feedback GAN (FBGAN) for DNA: a Novel Feedback-Loop Architecture for Optimizing Protein Functions
• Data Augmentation using Pre-trained Transformer Models
• DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome
• Evaluating Large Language Models Trained on Code
• Efficiently Modeling Long Sequences with Structured State Spaces
• The complete sequence of a human genome
• Biological Sequence Analysis
• LLaMA: Open and Efficient Foundation Language Models
• A draft human pangenome reference
• RWKV: Reinventing RNNs for the Transformer Era
• DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Gemini: A Family of Highly Capable Multimodal Models
• Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
• Sequence modeling and design from molecular to genome scale with Evo

Discussions

• Pangenome-Informed Language Models for Synthetic Genome Sequence Generation doi.org/10.1101/2024... [bsky, 0 points, 0 comments]