2024/10/21 by Yuwei Wan, Tong Xie, Wan, Yuwei +9 · 1 citation
Biochemistry, Genetics and Molecular Biology · #Biomedical Text Mining and Ontologies #Computation and Language (cs.CL) #Databases (cs.DB) #FOS: Computer and information sciences
paper · pdf · doi:10.48550/arxiv.2410.16165
openalex publication_date 2024/10/21 · openalex created_date 2024/11/14 · openalex updated_date 2026/07/28
Exploring the predictive capabilities of language models in material science is an ongoing interest. This study investigates the application of language model embeddings to enhance material property prediction in materials science. By evaluating various contextual embedding methods and pre-trained models, including Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformers (GPT), we demonstrate that domain-specific models, particularly MatBERT significantly outperform general-purpose models in extracting implicit knowledge from compound names and material properties. Our findings reveal that information-dense embeddings from the third layer of MatBERT, combined with a context-averaging approach, offer the most effective method for capturing material-property relationships from the scientific literature. We also identify a crucial "tokenizer effect," highlighting the importance of specialized text processing techniques that preserve complete compound names while maintaining consistent token counts. These insights underscore the value of domain-specific training and tokenization in materials science applications and offer a promising pathway for accelerating the discovery and development of new materials through AI-driven approaches.