RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture

2024/01/16 by Angels Balaguer, Angels de Luis Balaguer, Balaguer, Angels +33 · 2 voices · 20 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2401.08406

openalex publication_date 2024/01/16 · arxiv published 2024/01/16 · openalex created_date 2024/01/18 · arxiv updated 2024/01/30 · openalex updated_date 2026/07/28

Abstract

There are two common ways in which developers are incorporating proprietary and domain-specific data when building applications of Large Language Models (LLMs): Retrieval-Augmented Generation (RAG) and Fine-Tuning. RAG augments the prompt with the external data, while fine-Tuning incorporates the additional knowledge into the model itself. However, the pros and cons of both approaches are not well understood. In this paper, we propose a pipeline for fine-tuning and RAG, and present the tradeoffs of both for multiple popular LLMs, including Llama2-13B, GPT-3.5, and GPT-4. Our pipeline consists of multiple stages, including extracting information from PDFs, generating questions and answers, using them for fine-tuning, and leveraging GPT-4 for evaluating the results. We propose metrics to assess the performance of different stages of the RAG and fine-Tuning pipeline. We conduct an in-depth study on an agricultural dataset. Agriculture as an industry has not seen much penetration of AI, and we study a potentially disruptive application - what if we could provide location-specific insights to a farmer? Our results show the effectiveness of our dataset generation pipeline in capturing geographic-specific knowledge, and the quantitative and qualitative benefits of RAG and fine-tuning. We see an accuracy increase of over 6 p.p. when fine-tuning the model and this is cumulative with RAG, which increases accuracy by 5 p.p. further. In one particular experiment, we also demonstrate that the fine-tuned model leverages information from across geographies to answer specific questions, increasing answer similarity from 47% to 72%. Overall, the results point to how systems built using LLMs can be adapted to respond and incorporate knowledge across a dimension that is critical for a specific industry, paving the way for further applications of LLMs in other industrial domains.

Cited by

• AgriRegion: Region-Aware Retrieval for High-Fidelity Agricultural Advice
• CoEdge-RAG: Optimizing Hierarchical Scheduling for Retrieval-Augmented LLMs in Collaborative Edge Computing
• From Personal to Collective: On the Role of Local and Global Memory in LLM Personalization
• Enterprise AI Must Enforce Participant-Aware Access Control
• Causal-Counterfactual RAG: The Integration of Causal-Counterfactual Reasoning into RAG
• PowerChain: A Verifiable Agentic AI System for Automating Distribution Grid Analyses
• DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence
• Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime
• Leveraging Synthetic Data for Question Answering with Multilingual LLMs in the Agricultural Domain
• GAIus: Combining Genai with Legal Clauses Retrieval for Knowledge-based Assistant
• Knowledge Augmented Finetuning Matters in both RAG and Agent Based Dialog Systems
• PDF Retrieval Augmented Question Answering
• A Practical Guide for Evaluating LLMs and LLM-Reliant Systems
• Building Models of Neurological Language
• LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs
• Optimizing Retrieval Augmented Generation for Object Constraint Language
• Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corpora
• Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments
• Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization
• AgMMU: A Comprehensive Agricultural Multimodal Understanding Benchmark
• The Multi-Round Diagnostic RAG Framework for Emulating Clinical Reasoning

Discussions

• RAG vs. Fine-Tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture [hn, 3 points, 1 comments]
• RAG vs. Fine-Tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture [hn, 1 points, 0 comments]

Related

• AgriGPT: a Large Language Model Ecosystem for Agriculture
• Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems
• Fine-Tuning or Fine-Failing? Debunking Performance Myths in Large Language Models
• ALoFTRAG: Automatic Local Fine Tuning for Retrieval Augmented Generation
• Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs