2023/10/25 by Pei Zhang, Zhang, Pei, Logan T. Kearney +9 · 3 citations
Computer Science · Materials Science · #Chemical Physics (physics.chem-ph) #Computational Drug Discovery Methods #FOS: Computer and information sciences #FOS: Physical sciences #Machine Learning (cs.LG) #Machine Learning in Materials Science
paper · pdf · doi:10.48550/arxiv.2310.16958
openalex publication_date 2023/10/25 · openalex created_date 2023/10/28 · openalex updated_date 2026/07/28
Transformer-based large language models have remarkable potential to accelerate design optimization for applications such as drug development and materials discovery. Self-supervised pretraining of transformer models requires large-scale datasets, which are often sparsely populated in topical areas such as polymer science. State-of-the-art approaches for polymers conduct data augmentation to generate additional samples but unavoidably incurs extra computational costs. In contrast, large-scale open-source datasets are available for small molecules and provide a potential solution to data scarcity through transfer learning. In this work, we show that using transformers pretrained on small molecules and fine-tuned on polymer properties achieve comparable accuracy to those trained on augmented polymer datasets for a series of benchmark prediction tasks.