2025/03/19 by Mohamed-Salim Aissi, Aissi, Mohamed Salim, Clémence Grislain +9 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Data Visualization and Analytics #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Robotics (cs.RO)
paper · pdf · doi:10.48550/arxiv.2503.15108
openalex publication_date 2025/03/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
While Large Language Models (LLMs) excel at reasoning on text and Vision-Language Models (VLMs) are highly effective for visual perception, applying those models for visual instruction-based planning remains a widely open problem. In this paper, we introduce VIPER, a novel framework for multimodal instruction-based planning that integrates VLM-based perception with LLM-based reasoning. Our approach uses a modular pipeline where a frozen VLM generates textual descriptions of image observations, which are then processed by an LLM policy to predict actions based on the task goal. We fine-tune the reasoning module using behavioral cloning and reinforcement learning, improving our agent's decision-making capabilities. Experiments on the ALFWorld benchmark show that VIPER significantly outperforms state-of-the-art visual instruction-based planners while narrowing the gap with purely text-based oracles. By leveraging text as an intermediate representation, VIPER also enhances explainability, paving the way for a fine-grained analysis of perception and reasoning components.