KnowsLM: A framework for evaluation of small language models for knowledge augmentation and humanised conversations

2025/04/06 by Chitranshu Harbola, Harbola, Chitranshu, Anupam Purwar +1 · 7 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems

paper · pdf · doi:10.48550/arxiv.2504.04569

openalex publication_date 2025/04/06 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

In the evolving landscape of conversational AI, generating concise, context-aware, and human-like dialogue using small and medium-sized language models (LLMs) remains a complex challenge. This study investigates the influence of LoRA rank, dataset scale, and prompt prefix design on both knowledge retention and stylistic alignment. While fine-tuning improves fluency and enables stylistic customization, its ability to integrate unseen knowledge is constrained -- particularly with smaller datasets. Conversely, RAG-augmented models, equipped to incorporate external documents at inference, demonstrated superior factual accuracy on out-of-distribution prompts, though they lacked the stylistic consistency achieved by fine-tuning. Evaluations by LLM-based judges across knowledge accuracy, conversational quality, and conciseness suggest that fine-tuning is best suited for tone adaptation, whereas RAG excels at real-time knowledge augmentation.

Cited by

• M-PACE: Mother Child Framework for Multimodal Compliance
• E-ARMOR: Edge case Assessment and Review of Multilingual Optical Character Recognition
• VLMs-in-the-Wild: Bridging the Gap Between Academic Benchmarks and Enterprise Reality
• Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
• Prescriptive Agents based on RAG for Automated Maintenance (PARAM)
• AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMs
• A Survey on Collaborative Mechanisms Between Large and Small Language Models

Related

• User-Controlled Knowledge Fusion in Large Language Models: Balancing Creativity and Hallucination
• Should We Fine-Tune or RAG? Evaluating Different Techniques to Adapt LLMs for Dialogue
• StyleChat: Learning Recitation-Augmented Memory in LLMs for Stylized Dialogue Generation
• Efficient Tuning of Large Language Models for Knowledge-Grounded Dialogue Generation
• Leveraging LLMs for Dialogue Quality Measurement