Stress-Testing Multimodal Foundation Models for Crystallographic Reasoning

2025/06/16 by Can Polat, Polat, Can, Hasan Kurban +5 · 1 citation
Materials Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #FOS: Physical sciences #Machine Learning (cs.LG) #Machine Learning in Materials Science #Materials Science (cond-mat.mtrl-sci)

paper · pdf · doi:10.48550/arxiv.2506.13051

openalex publication_date 2025/06/16 · openalex created_date 2025/10/13 · openalex updated_date 2026/07/28

Abstract

Evaluating foundation models for crystallographic reasoning requires benchmarks that isolate generalization behavior while enforcing physical constraints. This work introduces a multiscale multicrystal dataset with two physically grounded evaluation protocols to stress-test multimodal generative models. The Spatial-Exclusion benchmark withholds all supercells of a given radius from a diverse dataset, enabling controlled assessments of spatial interpolation and extrapolation. The Compositional-Exclusion benchmark omits all samples of a specific chemical composition, probing generalization across stoichiometries. Nine vision--language foundation models are prompted with crystallographic images and textual context to generate structural annotations. Responses are evaluated via (i) relative errors in lattice parameters and density, (ii) a physics-consistency index penalizing volumetric violations, and (iii) a hallucination score capturing geometric outliers and invalid space-group predictions. These benchmarks establish a reproducible, physically informed framework for assessing generalization, consistency, and reliability in large-scale multimodal models. Dataset and code are available at https://github.com/KurbanIntelligenceLab/StressTestingMMFMinCR.

Citations

• xChemAgents: Agentic AI for Explainable Quantum Chemistry
• MatterChat: A Multi-Modal LLM for Material Science
• HoneyComb: A Flexible LLM-Based Agent System for Materials Science
• ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area
• LAB-Bench: Measuring Capabilities of Language Models for Biology Research
• Are LLMs Ready for Real-World Materials Discovery?
• Multimodal machine learning for materials science: composition-structure bimodal learning for experimentally measured properties
• EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations
• FAENet: Frame Averaging Equivariant GNN for Materials Modeling
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
• Directional Message Passing for Molecular Graphs
• MoleculeNet: A Benchmark for Molecular Machine Learning
• Self-Consistent Equations Including Exchange and Correlation Effects

Cited by

• AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials
• ChronoVision: Temporal Reasoning via Latent State Reconstruction

Related

• LeMat-GenBench: A Unified Evaluation Framework for Crystal Generative Models
• AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials
• XxaCT-NN: Structure Agnostic Multimodal Learning for Materials Science
• CrysMMNet: Multimodal Representation for Crystal Property Prediction
• Scalable Diffusion for Materials Generation