Design and Evaluation of a Retrieval-Augmented Generation Architecture for OWASP Security Data

2023/09/26 by Es, Shahul, Jithin James, Luis Espinosa-Anke +4 · 2 voices · 61 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2309.15217

openalex publication_date 2023/09/26 · openalex created_date 2023/09/30 · openalex updated_date 2026/07/28

Abstract

Project Description This archive contains a research artifact for the design, implementation, and evaluation of a Retrieval-Augmented Generation (RAG) system in the domain of application security. The artifact provides a complete, self-contained snapshot of the system architecture, curated evaluation data, and automated benchmarking pipelines. RAG System Architecture The archive includes a fully implemented RAG pipeline designed to ground large language model outputs in authoritative security documentation. The system follows a modular architecture with clearly separated components for document ingestion, semantic retrieval, and answer generation. The knowledge base is derived from OWASP Cheat Sheet Series and related security guidance, enabling security-focused question answering with reduced risk of hallucinated responses. Ground Truth Dataset A ground truth dataset is included to support systematic evaluation. The dataset consists of security-relevant questions and corresponding reference information derived from OWASP materials. It is intended exclusively for offline evaluation and benchmarking of RAG behavior and is not meant to serve as training data.

Cited by

• DICE: Discrete Interpretable Comparative Evaluation with Probabilistic Scoring for Retrieval-Augmented Generation
• ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents
• Auditing Institutional Heterogeneity for Generative AI in Patient Education: A Large-Scale Study of 102 US Transplant Handbooks
• Agentic Graph Retrieval-Augmented Generation for Auditable Commercial Registry Analysis
• The Meta-Prompting Protocol: Orchestrating LLMs via Adversarial Feedback Loops
• Hybrid Retrieval-Augmented Generation for Robust Multilingual Document Question Answering
• Toward Faithful Retrieval-Augmented Generation with Sparse Autoencoders
• AI Application in Anti-Money Laundering for Sustainable and Transparent Financial Systems
• Breaking It Down: Domain-Aware Semantic Segmentation for Retrieval Augmented Generation
• Learning to Debug: LLM-Organized Knowledge Trees for Solving RTL Assertion Failures
• CARE-RAG - Clinical Assessment and Reasoning in RAG
• Build AI Assistants using Large Language Models and Agents to Enhance the Engineering Education of Biomechanics
• LiveRAG: A diverse Q&A dataset with varying difficulty level for RAG evaluation
• A Graph-based RAG for Energy Efficiency Question Answering
• RAGSmith: A Framework for Finding the Optimal Composition of Retrieval-Augmented Generation Methods Across Datasets
• BridgeEQA: Virtual Embodied Agents for Real Bridge Inspections
• Large Language Models for Explainable Threat Intelligence
• RAGalyst: Automated Human-Aligned Agentic Evaluation for Domain-Specific RAG
• Advancing Subsurface Discovery and Geothermal Monitoring with an Agentic Artificial Intelligence Framework
• Adapting Large Language Models to Emerging Cybersecurity using Retrieval Augmented Generation
• OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora
• GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation
• Evaluation of Agents under Simulated AI Marketplace Dynamics
• Chitchat with AI: Understand the supply chain carbon disclosure of companies worldwide through Large Language Model
• InterpDetect: Interpretable Signals for Detecting Hallucinations in Retrieval-Augmented Generation
• AtlasKV: Augmenting LLMs with Billion-Scale Knowledge Graphs in 20GB VRAM
• Disparities in Multilingual LLM-Based Healthcare Q&A
• Demo: Guide-RAG: Evidence-Driven Corpus Curation for Retrieval-Augmented Generation in Long COVID
• Knowledge-Graph Based RAG System Evaluation Framework
• A survey on retrieval-augmentation generation (RAG) models for healthcare applications
• VeritasFi: An Adaptable, Multi-tiered RAG Framework for Multi-modal Financial Question Answering
• FATHOMS-RAG: A Framework for the Assessment of Thinking and Observation in Multimodal Systems that use Retrieval Augmented Generation
• IoDResearch: Deep Research on Private Heterogeneous Data via the Internet of Data
• Exposing Citation Vulnerabilities in Generative Engines
• Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelines
• Data Quality Challenges in Retrieval-Augmented Generation
• Auto-ARGUE: LLM-Based Report Generation Evaluation
• Think Less, Label Better: Multi-Stage Domain-Grounded Synthetic Data Generation for Fine-Tuning Large Language Models in Telecommunications
• Open Security Benchmark: Towards Autonomous Enterprise Cyber Defense
• LayerRAG-Bench: A Cross-Layer Reliability Benchmark for Agentic Retrieval-Augmented Generation
• <i>ToxTempAssistant</i> : using large language models to standardise cell-based toxicological test method descriptions
• Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG
• SSFO: Self-Supervised Faithfulness Optimization for Retrieval-Augmented Generation
• Large Language Models for Security Operations Centers: A Comprehensive Survey
• Investigating Student Interaction Patterns with Large Language Model-Powered Course Assistants in Computer Science Courses
• DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence
• OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviews
• Reasoning-Intensive Regression
• An Agile Method for Implementing Retrieval Augmented Generation Tools in Industrial SMEs
• From Search to Reasoning: A Five-Level RAG Capability Framework for Enterprise Data
• Retrieval-Augmented Generation for Natural Language Art Provenance Searches in the Getty Provenance Index
• Flexible metadata harvesting for ecology using large language models
• FALCON: Autonomous Cyber Threat Intelligence Mining with LLMs for IDS Rule Generation
• Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap
• GeoGPT-RAG Technical Report
• Can we Evaluate RAGs with Synthetic Data?
• Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluation
• Multi-Modal Requirements Data-based Acceptance Criteria Generation using LLMs
• RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation
• Towards a rigorous evaluation of RAG systems: the challenge of due diligence
• Evaluation and Benchmarking of LLM Agents: A Survey

Discussions

• Ragas: Automated Evaluation of Retrieval Augmented Generation [hn, 4 points, 0 comments]
• Ragas: Automated Evaluation of Retrieval Augmented Generation arxiv.org/abs/2309.15217 [bsky, 0 points, 0 comments]

Related

• RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation
• RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation
• Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
• RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
• BERGEN: A Benchmarking Library for Retrieval-Augmented Generation