Towards Requirements Engineering for RAG Systems

2025/05/12 by Tor Sporsem, Sporsem, Tor, Rasmus Ulfsnes +1
Computer Science · #AI-based Problem Solving and Planning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Persona Design and Applications #Software Engineering (cs.SE) #Software Engineering Techniques and Practices

paper · pdf · doi:10.48550/arxiv.2505.07553

openalex publication_date 2025/05/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This short paper explores how a maritime company develops and integrates large-language models (LLM). Specifically by looking at the requirements engineering for Retrieval Augmented Generation (RAG) systems in expert settings. Through a case study at a maritime service provider, we demonstrate how data scientists face a fundamental tension between user expectations of AI perfection and the correctness of the generated outputs. Our findings reveal that data scientists must identify context-specific "retrieval requirements" through iterative experimentation together with users because they are the ones who can determine correctness. We present an empirical process model describing how data scientists practically elicited these "retrieval requirements" and managed system limitations. This work advances software engineering knowledge by providing insights into the specialized requirements engineering processes for implementing RAG systems in complex domain-specific applications.

Citations

• RAG based Question-Answering for Contextual Response Prediction System
• Teaching Theorizing in Software Engineering Research
• Naming the Pain in Machine Learning-Enabled Systems Engineering
• Transforming Software Development with Generative AI: Empirical Insights on Collaboration and Workflow
• Seven Failure Points When Engineering a Retrieval Augmented Generation System
• Benchmarking Large Language Models in Retrieval-Augmented Generation
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• Requirements Practices and Gaps When Engineering Human-Centered Artificial Intelligence Systems
• Is AI Ground Truth Really True? The Dangers of Training and Evaluating AI Tools Based on Experts’ Know-What
• A Software Engineering Perspective on Engineering Machine Learning\n Systems: State of the Art and Challenges
• A software engineering perspective on engineering machine learning systems: State of the art and challenges
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Requirements Engineering for Machine Learning: Perspectives from Data\n Scientists
• The Mythos of Model Interpretability

Related

• Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation
• Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluation
• RAG Does Not Work for Enterprises
• Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective
• Seven Failure Points When Engineering a Retrieval Augmented Generation System