PRISM: Perception Reasoning Interleaved for Sequential Decision Making

2026/05/06 by Mohamed Salim Aissi, Mohamed-Salim Aissi, Clémence Grislain +6 · 1 voice
Computer Science · Psychology · #Embodied cognition #Multimodal Machine Learning Applications #Perception #Pipeline (software) #Prism #Scaling #Social Robot Interaction and HRI #Topic Modeling #Work (physics) #cs.AI

paper · pdf · open access · doi:10.48550/arxiv.2605.05407

published in HAL (Le Centre pour la Communication Scientifique Directe) (Centre National de la Recherche Scientifique)

openalex publication_date 2026/05/06 · arxiv published 2026/05/06 · openalex created_date 2026/05/09 · arxiv updated 2026/06/12 · openalex updated_date 2026/07/30

Abstract

Scaling LLM-based embodied agents from text-only environments to complex multimodal settings remains a major challenge. Recent work identifies a perception-reasoning-decision gap in standalone Vision-Language Models (VLMs), which often overlook task-critical information. In this paper, we introduce PRISM, a framework that tightly couples perception (VLM) and decision (LLM) through a dynamic question-answer (DQA) pipeline. Instead of passively accepting the VLM's description, the LLM critiques it, probes the VLM with goal-oriented questions, and synthesizes a compact image description. This closed-loop interaction yields a sharp, task-driven understanding of the scene. We evaluate PRISM on the ALFWorld and Room-to-Room (R2R) benchmarks. We show that: (1) PRISM significantly outperforms state-of-the-art image-based models, (2) our Interactive goal-oriented perception pipeline yields systematic and substantial gains, and (3) PRISM is fully automatic, eliminating the need for handcrafted questions or answers.

Citations

• Vision-and-Language Navigation with Analogical Textual Descriptions in LLMs
• VIPER: Visual Perception and Explainable Reasoning for Sequential Decision-Making
• Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
• What matters when building vision-language models?
• Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions
• Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld
• LangNav: Language as a Perceptual Representation for Navigation
• Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions
• NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• Asking Before Acting: Gather Information in Embodied Decision Making with Language Models
• Evaluating Object Hallucination in Large Vision-Language Models
• Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents
• Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
• A Framework for Learning to Request Rich and Contextually Useful Information from Humans
• LoRA: Low-Rank Adaptation of Large Language Models
• ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
• BERTScore: Evaluating Text Generation with BERT
• Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments
• Proximal Policy Optimization Algorithms

Discussions

• We are very proud to share that the team will be present at ICML this year again! 1. "PRISM: Perception Reasoning Interleaved for Sequential Decision Making". arxiv.org/abs/2605.05407 2. "Improving Ze [bsky, 1 points, 0 comments]

Related

• Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
• Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond
• SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning In Text-only LLMs
• VIPER: Visual Perception and Explainable Reasoning for Sequential Decision-Making
• Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs