MORPH: PDE Foundation Models with Arbitrary Data Modality

2025/09/25 by Mahindra Rautela, Alexander Most, Rautela, Mahindra Singh +11
Computer Science · Materials Science · Physics and Astronomy · #Artificial Intelligence (cs.AI) #Computational Physics (physics.comp-ph) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Physical sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning in Materials Science #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.2509.21670

openalex publication_date 2025/09/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30

Abstract

We introduce MORPH, a modality-agnostic, autoregressive foundation model for partial differential equations (PDEs). MORPH is built on a convolutional vision transformer backbone that seamlessly handles heterogeneous spatiotemporal datasets of varying data modality (1D--3D) at different resolutions, and multiple fields with mixed scalar and vector components. The architecture combines (i) component-wise convolution, which jointly processes scalar and vector channels to capture local interactions, (ii) inter-field cross-attention, which models and selectively propagates information between different physical fields, (iii) axial attentions, which factorize full spatiotemporal self-attention along individual spatial and temporal axes to reduce computational burden while retaining expressivity. We pretrain multiple model variants on a diverse collection of heterogeneous PDE datasets and evaluate transfer to a range of downstream prediction tasks. Using both full-model fine-tuning and parameter-efficient low-rank adapters, MORPH outperforms models trained from scratch. Across extensive evaluations, MORPH matches or surpasses strong baselines and recent state-of-the-art models. Collectively, these capabilities present a flexible and powerful backbone for learning from the heterogeneous and multimodal nature of scientific observations, charting a path toward scalable and data-efficient scientific machine learning. The source code, datasets, and models are publicly available at https://github.com/lanl/MORPH.

Citations

• PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations
• DISCO: learning to DISCover an evolution Operator for multi-physics-agnostic prediction
• The Well: a Large-Scale Collection of Diverse Physics Simulations for Machine Learning
• PROSE-FD: A Multimodal PDE Foundation Model for Learning Multiple Operators for Forecasting Fluid Dynamics
• Spatially-Aware Diffusion Models with Cross-Attention for Global Field Reconstruction with Sparse Observations
• Generative Learning of the Solution of Parametric Partial Differential Equations Using Guided Diffusion Models and Virtual Observations
• Poseidon: Efficient Foundation Models for PDEs
• A conditional latent autoregressive recurrent model for generation and forecasting of beam dynamics in particle accelerators
• DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training
• Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning
• PDEformer: Towards a Foundation Model for One-Dimensional Partial Differential Equations
• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
• CFDBench: A Large-Scale Benchmark for Machine Learning Methods in Fluid Dynamics
• β-Variational autoencoders and transformers for reduced-order modelling of fluid flows
• 3D-Spatiotemporal Forecasting the Expansion of Supernova Shells Using Deep Learning toward High-Resolution Galaxy Simulations
• ClimaX: A foundation model for weather and climate
• Transfer Learning Enhanced DeepONet for Long-Time Prediction of Evolution Equations
• PDEBENCH: An Extensive Benchmark for Scientific Machine Learning
• Towards Multi-spatiotemporal-scale Generalized PDE Modeling
• Transformer for Partial Differential Equations' Operator Learning
• U-NO: U-shaped Neural Operators
• Learning two-phase microstructure evolution using neural operators and autoencoder architectures
• Physics-Informed Neural Operator for Learning Partial Differential Equations
• LoRA: Low-Rank Adaptation of Large Language Models
• Intrinsic Dimensionality Explains the Effectiveness of Language Model\n Fine-Tuning
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Fourier Neural Operator for Parametric Partial Differential Equations
• Hierarchical Deep Learning of Multiscale Differential Equation Time-Steppers
• Multiscale Simulations of Complex Systems by Learning their Effective Dynamics
• Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators
• Axial Attention in Multidimensional Transformers
• Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• Neural Ordinary Differential Equations
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• U-Net: Convolutional Networks for Biomedical Image Segmentation

Related

• UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation
• Towards a Foundation Model for Partial Differential Equations: Multi-Operator Learning and Extrapolation
• Towards a Foundation Model for Partial Differential Equations Across Physics Domains
• PI-MFM: Physics-informed multimodal foundation model for solving partial differential equations
• Enhanced accuracy through ensembling of randomly initialized auto-regressive models for time-dependent PDEs