Retrieval-augmented reasoning with lean language models

2025/08/15 by Chan, Ryan Sze-Yin, Nanni, Federico, Lazauskas, Tomas +6 · 1 citation
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computers and Society (cs.CY) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2508.11386

Abstract

This technical report details a novel approach to combining reasoning and retrieval augmented generation (RAG) within a single, lean language model architecture. While existing RAG systems typically rely on large-scale models and external APIs, our work addresses the increasing demand for performant and privacy-preserving solutions deployable in resource-constrained or secure environments. Building on recent developments in test-time scaling and small-scale reasoning models, we develop a retrieval augmented conversational agent capable of interpreting complex, domain-specific queries using a lightweight backbone model. Our system integrates a dense retriever with fine-tuned Qwen2.5-Instruct models, using synthetic query generation and reasoning traces derived from frontier models (e.g., DeepSeek-R1) over a curated corpus, in this case, the NHS A-to-Z condition pages. We explore the impact of summarisation-based document compression, synthetic data design, and reasoning-aware fine-tuning on model performance. Evaluation against both non-reasoning and general-purpose lean models demonstrates that our domain-specific fine-tuning approach yields substantial gains in answer accuracy and consistency, approaching frontier-level performance while remaining feasible for local deployment. All implementation details and code are publicly released to support reproducibility and adaptation across domains.

Citations

• Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
• Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning
• Small Language Models are the Future of Agentic AI
• Qwen3 Technical Report
• TTRL: Test-Time Reinforcement Learning
• Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
• A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
• Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling
• s1: Simple test-time scaling
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness
• Prompto: An open source library for asynchronous querying of LLM endpoints
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• The Llama 3 Herd of Models
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
• Gemma 2: Improving Open Language Models at a Practical Size
• On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models
• A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
• Isambard-AI: a leadership class supercomputer optimised specifically for Artificial Intelligence
• Reliable, Adaptable, and Attributable Language Models with Retrieval
• Enhancing Recommendation Diversity by Re-ranking with Large Language Models
• The Faiss library
• Retrieval-Augmented Generation for Large Language Models: A Survey
• Universal Self-Consistency for Large Language Model Generation
• MemGPT: Towards LLMs as Operating Systems
• Language Models as a Service: Overview of a New Paradigm and its Challenges
• Qwen Technical Report
• The Rise and Potential of Large Language Model Based Agents: A Survey
• A Simple and Effective Pruning Approach for Large Language Models
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
• Reflexion: Language Agents with Verbal Reinforcement Learning
• LLaMA: Open and Efficient Foundation Language Models
• In-Context Retrieval-Augmented Language Models
• REPLUG: Retrieval-Augmented Black-Box Language Models
• Teaching Small Language Models to Reason
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Distilling Knowledge from Reader to Retriever for Question Answering
• Answering Any-hop Open-domain Questions with Iterative Document Reranking
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Dense Passage Retrieval for Open-Domain Question Answering
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Quantization and Training of Neural Networks for Efficient\n Integer-Arithmetic-Only Inference
• Distilling the Knowledge in a Neural Network
• Sequence Transduction with Recurrent Neural Networks
• OpenAI o1 System Card

Cited by

• Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMs

Related

• Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large Language Models
• Fishing for Answers: Exploring One-shot vs. Iterative Retrieval Strategies for Retrieval Augmented Generation
• Retrieval-Augmented Generation: A Comprehensive Survey of Architectures, Enhancements, and Robustness Frontiers
• Improving Context Fidelity via Native Retrieval-Augmented Reasoning
• A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions