LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Engines

2024/11/12 by Anoop Cherian, Cherian, Anoop, Radu Corcodel +5 · 9 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Robotics (cs.RO) #Semantic Web and Ontologies #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2411.08027

openalex publication_date 2024/11/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Most learning-based approaches to complex physical reasoning sidestep the crucial problem of parameter identification (e.g., mass, friction) that governs scene dynamics, despite its importance in real-world applications such as collision avoidance and robotic manipulation. In this paper, we present LLMPhy, a black-box optimization framework that integrates large language models (LLMs) with physics simulators for physical reasoning. The core insight of LLMPhy is to bridge the textbook physical knowledge embedded in LLMs with the world models implemented in modern physics engines, enabling the construction of digital twins of input scenes via latent parameter estimation. Specifically, LLMPhy decomposes digital twin construction into two subproblems: (i) a continuous problem of estimating physical parameters and (ii) a discrete problem of estimating scene layout. For each subproblem, LLMPhy iteratively prompts the LLM to generate computer programs encoding parameter estimates, executes them in the physics engine to reconstruct the scene, and uses the resulting reconstruction error as feedback to refine the LLM's predictions. As existing physical reasoning benchmarks rarely account for parameter identifiability, we introduce three new datasets designed to evaluate physical reasoning in zero-shot settings. Our results show that LLMPhy achieves state-of-the-art performance on our tasks, recovers physical parameters more accurately, and converges more reliably than prior black-box methods. See the LLMPhy project page for details: https://www.merl.com/research/highlights/LLMPhy

Cited by

• 4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models
• Keeping it Local, Tiny and Real: Automated Report Generation on Edge Computing Devices for Mechatronic-Based Cognitive Systems
• TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibility
• InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoning
• Teaching Physical Awareness to LLMs through Sounds
• G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibration
• RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills
• Can Theoretical Physics Research Benefit from Language Agents?
• GOBench: Benchmarking Geometric Optics Generation and Understanding of MLLMs

Related

• Inducing Causal World Models in LLMs for Zero-Shot Physical Reasoning
• NEWTON: Are Large Language Models Capable of Physical Reasoning?
• A little less conversation, a little more action, please: Investigating the physical common-sense of LLMs in a 3D embodied environment
• SimLM: Can Language Models Infer Parameters of Physical Systems?
• Physics Context Builders: A Modular Framework for Physical Reasoning in Vision-Language Models