Project Riley: Multimodal Multi-Agent LLM Collaboration with Emotional Reasoning and Voting

2025/05/26 by Ana Rita Ortigoso, Ortigoso, Ana Rita, Gabriel Vieira +9 · 2 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #H.5.2 #I.2.1 #I.2.7 #Multi-Agent Systems and Negotiation #Natural Language Processing Techniques #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2505.20521

openalex publication_date 2025/05/26 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

This paper presents Project Riley, a novel multimodal and multi-model conversational AI architecture oriented towards the simulation of reasoning influenced by emotional states. Drawing inspiration from Pixar's Inside Out, the system comprises five distinct emotional agents - Joy, Sadness, Fear, Anger, and Disgust - that engage in structured multi-round dialogues to generate, criticise, and iteratively refine responses. A final reasoning mechanism synthesises the contributions of these agents into a coherent output that either reflects the dominant emotion or integrates multiple perspectives. The architecture incorporates both textual and visual large language models (LLMs), alongside advanced reasoning and self-refinement processes. A functional prototype was deployed locally in an offline environment, optimised for emotional expressiveness and computational efficiency. From this initial prototype, another one emerged, called Armando, which was developed for use in emergency contexts, delivering emotionally calibrated and factually accurate information through the integration of Retrieval-Augmented Generation (RAG) and cumulative context tracking. The Project Riley prototype was evaluated through user testing, in which participants interacted with the chatbot and completed a structured questionnaire assessing three dimensions: Emotional Appropriateness, Clarity and Utility, and Naturalness and Human-likeness. The results indicate strong performance in structured scenarios, particularly with respect to emotional alignment and communicative clarity.

Citations

• EmoDebt: Bayesian-Optimized Emotional Intelligence for Strategic Agent-to-Agent Debt Recovery
• Exploring Emotion-Sensitive LLM-Based Conversational AI
• MALT: Improving Reasoning with Multi-Agent LLM Training
• An Electoral Approach to Diversify LLM-based Multi-Agent Collective Decision-Making
• Emotion-Aware Embedding Fusion in LLMs (Flan-T5, LLAMA 2, DeepSeek-R1, and ChatGPT 4) for Intelligent Response Generation
• Nadine: An LLM-driven Intelligent Social Robot with Affective Capabilities and Human-like Memory
• Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration
• ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
• A psychoevolutionary theory of emotions

Cited by

• Agent-Based Modular Learning for Multimodal Emotion Recognition in Human-Agent Systems
• Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives

Related

• AI with Emotions: Exploring Emotional Expressions in Large Language Models
• Intelligent Agents with Emotional Intelligence: Current Trends, Challenges, and Future Prospects
• Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark
• When LLMs Team Up: The Emergence of Collaborative Affective Computing
• Emotion Twenty Questions Dialog System for Lexical Emotional Intelligence