Distillation of atomistic foundation models across architectures and chemical domains

2025/06/12 by John L. A. Gardner, Daniel F. Thomas du Toit, Gardner, John L. A. +19 · 3 voices · 3 citations
#physics.comp-ph

paper · pdf · doi:10.48550/arxiv.2506.10956

Abstract

Machine-learned interatomic potentials have transformed computational research in the physical sciences. Recent atomistic `foundation' models have changed the field yet again: trained on many different chemical elements and domains, these potentials are widely applicable, but comparably slow and resource-intensive to run. Here we show how distillation via synthetic data can be used to cheaply transfer knowledge from atomistic foundation models to a range of different architectures, unlocking much smaller, more efficient potentials. We demonstrate speed-ups of > 10× by distilling from one graph-network architecture into another, and > 100× by leveraging the atomic cluster expansion framework. We showcase applicability across chemical and materials domains: from liquid water to hydrogen under extreme conditions; from porous silica and a hybrid halide perovskite solar-cell material to modelling organic reactions. Our work shows how distillation can support the routine and computationally efficient use of current and future atomistic foundation models in real-world scientific research.

Citations

• Towards Fast, Specialized Machine Learning Force Fields: Distilling Foundation Models via Energy Hessians
• LAMMPS - a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales
• On the Opportunities and Risks of Foundation Models
• Language Models are Few-Shot Learners
• Distilling the Knowledge in a Neural Network
• PLUMED 2: New feathers for an old bird
• A Stochastic Approximation Method

Cited by

• Design, Assessment, and Application of Machine Learning Potential Energy Surfaces
• Performance of universal machine-learned potentials with explicit long-range interactions in biomolecular simulations
• Heterogeneous Ensemble Enables a Universal Uncertainty Metric for Atomistic Foundation Models

Discussions

• Distilling atomistic foundation models! ⚗️🧪🤖 In this #compchem preprint, we describe a general (“architecture-agnostic”) approach to creating fast, application-specific MLIP models via synthetic dat [bsky, 17 points, 0 comments]
• 3) There is a new distillation method and code (arxiv.org/abs/2506.10956) that accelerates universal NNPs by 10 - 100 x. [bsky, 1 points, 1 comments]
• For (many) more results, please see the pre-print! arxiv.org/abs/2506.10956 [bsky, 0 points, 1 comments]

Related

• Foundation Models for Atomistic Simulation of Chemistry and Materials
• Graph atomic cluster expansion for foundational machine learning interatomic potentials
• Data-model Coevolution as the Architectural Principle for AI-Native Materials Databases
• A foundation model for atomistic materials chemistry
• Transformer Atomic Cluster Expansion: TRACE