GrowOVER: How Can LLMs Adapt to Growing Real-World Knowledge?

2024/06/09 by Dayoon Ko, Jinyoung Kim, Ko, Dayoon +5 · 3 citations
Computer Science · #Semantic Web and Ontologies

paper · pdf · doi:10.48550/arxiv.2406.05606

Abstract

In the real world, knowledge is constantly evolving, which can render existing knowledge-based datasets outdated. This unreliability highlights the critical need for continuous updates to ensure both accuracy and relevance in knowledge-intensive tasks. To address this, we propose GrowOVER-QA and GrowOVER-Dialogue, dynamic open-domain QA and dialogue benchmarks that undergo a continuous cycle of updates, keeping pace with the rapid evolution of knowledge. Our research indicates that retrieval-augmented language models (RaLMs) struggle with knowledge that has not been trained on or recently updated. Consequently, we introduce a novel retrieval-interactive language model framework, where the language model evaluates and reflects on its answers for further re-retrieval. Our exhaustive experiments demonstrate that our training-free framework significantly improves upon existing methods, performing comparably to or even surpassing continuously trained language models.

Cited by

• Look It Up: Analysing Internal Web Search Capabilities of Modern LLMs
• ChronoPlay: A Framework for Modeling Dual Dynamics and Authenticity in Game RAG Benchmarks
• MegaWika 2: A More Comprehensive Multilingual Collection of Articles and their Sources

Related

• Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs
• Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models
• EvoWiki: Evaluating LLMs on Evolving Knowledge
• OKBench: Democratizing LLM Evaluation with Fully Automated, On-Demand, Open Knowledge Benchmarking
• Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models