From Templates to Natural Language: Generalization Challenges in Instruction-Tuned LLMs for Spatial Reasoning

2025/05/20 by Chalamalasetti Kranti, Kranti, Chalamalasetti, Sherzod Hakimov +3
Computer Science · #AI-based Problem Solving and Planning #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2505.14425

openalex publication_date 2025/05/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Instruction-tuned large language models (LLMs) have shown strong performance on a variety of tasks; however, generalizing from synthetic to human-authored instructions in grounded environments remains a challenge for them. In this work, we study generalization challenges in spatial grounding tasks where models interpret and translate instructions for building object arrangements on a 2.5D grid. We fine-tune LLMs using only synthetic instructions and evaluate their performance on a benchmark dataset containing both synthetic and human-written instructions. Our results reveal that while models generalize well on simple tasks, their performance degrades significantly on more complex tasks. We present a detailed error analysis of the gaps in instruction generalization.

Citations

• REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?
• Synthetic Data Generation Using Large Language Models: Advances in Text and Code
• Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models
• Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Reset
• Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming
• Instruction Following with Goal-Conditioned Reinforcement Learning in Virtual Environments
• Nebula: A discourse aware Minecraft Builder
• Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft
• Toward Automated Programming for Robotic Assembly Using ChatGPT
• VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
• Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents
• Task and Motion Planning with Large Language Models for Object Rearrangement
• ChatGPT for Robotics: Design Principles and Model Abilities
• On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)
• Visual Language Maps for Robot Navigation
• Code as Policies: Language Model Programs for Embodied Control
• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
• Draw Me a Flower: Processing and Grounding Abstraction in Natural Language
• Language Models are Few-Shot Learners
• Experience Grounds Language
• ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday\n Tasks
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Interactive Visual Grounding of Referring Expressions for Human-Robot\n Interaction
• Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions

Related

• Grounding Natural Language Instructions: Can Large Language Models Capture Spatial Information?
• Evaluating Spatial Understanding of Large Language Models
• Can Large Language Models Create New Knowledge for Spatial Reasoning Tasks?
• Stuck in the Matrix: Probing Spatial Reasoning in Large Language Models
• Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark