Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments

2025/05/02 by Bolton, Regan, Sheikhfathollahi, Mohammadreza, Parkinson, Simon +4 · 1 citation
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Software Engineering (cs.SE)

paper · doi:10.48550/arxiv.2505.01307

Abstract

Safety critical software assessment requires robust assessment against complex regulatory frameworks, a process traditionally limited by manual evaluation. This paper presents Document Retrieval-Augmented Fine-Tuning (DRAFT), a novel approach that enhances the capabilities of a large language model (LLM) for safety-critical compliance assessment. DRAFT builds upon existing Retrieval-Augmented Generation (RAG) techniques by introducing a novel fine-tuning framework that accommodates our dual-retrieval architecture, which simultaneously accesses both software documentation and applicable reference standards. To fine-tune DRAFT, we develop a semi-automated dataset generation methodology that incorporates variable numbers of relevant documents with meaningful distractors, closely mirroring real-world assessment scenarios. Experiments with GPT-4o-mini demonstrate a 7% improvement in correctness over the baseline model, with qualitative improvements in evidence handling, response structure, and domain-specific reasoning. DRAFT represents a practical approach to improving compliance assessment systems while maintaining the transparency and evidence-based reasoning essential in regulatory domains.

Citations

• GoRA: Gradient-driven Adaptive Low Rank Adaptation
• LLM-based event log analysis techniques: A survey
• SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation
• GPT-4o System Card
• Hybrid RAG-empowered Multi-modal LLM for Secure Data Management in Internet of Medical Things: A Diffusion-based Contract Approach
• UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis
• RAFT: Adapting Language Model to Domain Specific RAG
• Retrieval-Augmented Generation for AI-Generated Content: A Survey
• When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method
• LoRA+: Efficient Low Rank Adaptation of Large Models
• ChatQA: Surpassing GPT-4 on Conversational QA and RAG
• RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture
• Seven Failure Points When Engineering a Retrieval Augmented Generation System
• Retrieval-Augmented Generation for Large Language Models: A Survey
• LoRA: Low-Rank Adaptation of Large Language Models
• Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
• Language Models are Few-Shot Learners
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Cited by

• AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliance
• Multi-Stage Retrieval for Operational Technology Cybersecurity Compliance Using Large Language Models: A Railway Casestudy

Related

• RAGuard: A Novel Approach for in-context Safe Retrieval Augmented Generation for LLMs
• VERA: Validation and Evaluation of Retrieval-Augmented Systems
• Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?
• Trustworthiness in Retrieval-Augmented Generation Systems: A Survey
• RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models