FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation

2023/10/05 by Tu Vu, Mohit Iyyer, Vu, Tu +20 · 3 voices · 88 citations
Computer Science · #Expert finding and Q&A systems #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2310.03214

openalex publication_date 2023/10/05 · openalex created_date 2023/10/09 · openalex updated_date 2026/07/28

Abstract

Most large language models (LLMs) are trained once and never updated; thus, they lack the ability to dynamically adapt to our ever-changing world. In this work, we perform a detailed study of the factuality of LLM-generated text in the context of answering questions that test current world knowledge. Specifically, we introduce FreshQA, a novel dynamic QA benchmark encompassing a diverse range of question and answer types, including questions that require fast-changing world knowledge as well as questions with false premises that need to be debunked. We benchmark a diverse array of both closed and open-source LLMs under a two-mode evaluation procedure that allows us to measure both correctness and hallucination. Through human evaluations involving more than 50K judgments, we shed light on limitations of these models and demonstrate significant room for improvement: for instance, all models (regardless of model size) struggle on questions that involve fast-changing knowledge and false premises. Motivated by these results, we present FreshPrompt, a simple few-shot prompting method that substantially boosts the performance of an LLM on FreshQA by incorporating relevant and up-to-date information retrieved from a search engine into the prompt. Our experiments show that FreshPrompt outperforms both competing search engine-augmented prompting methods such as Self-Ask (Press et al., 2022) as well as commercial systems such as Perplexity.AI. Further analysis of FreshPrompt reveals that both the number of retrieved evidences and their order play a key role in influencing the correctness of LLM-generated answers. Additionally, instructing the LLM to generate concise and direct answers helps reduce hallucination compared to encouraging more verbose answers. To facilitate future work, we release FreshQA at github.com/freshllms/freshqa and commit to updating it at regular intervals.

Cited by

• AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learning
• Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models
• The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality
• Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers
• CryptoBench: A Dynamic Benchmark for Expert-Level Evaluation of LLM Agents in Cryptocurrency
• "AGI" team at SHROOM-CAP: Data-Centric Approach to Multilingual Hallucination Detection using XLM-RoBERTa
• "To Survive, I Must Defect": Jailbreaking LLMs via the Game-Theory Scenarios
• HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models
• Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation
• Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective
• InfoFlow: Reinforcing Search Agent Via Reward Density Optimization
• Argument Collapse: LLMs Flatten Long-Form Public Debate
• Prior Knowledge Makes It Possible: From Sublinear Graph Algorithms to LLM Test-Time Methods
• Judging Against the Reference: Uncovering Knowledge-Driven Failures in LLM-Judges on QA Evaluation
• Mitigating Hallucination in Large Language Models (LLMs): An Application-Oriented Survey on RAG, Reasoning, and Agentic Systems
• SynthWorlds: Controlled Parallel Worlds for Disentangling Reasoning and Knowledge in Language Models
• MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs
• A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
• FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMs
• RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Models
• When Benchmarks Age: Temporal Misalignment through Large Language Model Factuality Evaluation
• Large Language Models Hallucination: A Comprehensive Survey
• SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents
• The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation From Recognition to Reasoning
• Speculating LLMs' Chinese Training Data Pollution from Their Tokens
• Improving Context Fidelity via Native Retrieval-Augmented Reasoning
• SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge
• Facts Fade Fast: Evaluating Memorization of Outdated Medical Knowledge in Large Language Models
• AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMs
• Re3: Learning to Balance Relevance & Recency for Temporal Information Retrieval
• Beyond Memorization: Reasoning-Driven Synthesis as a Mitigation Strategy Against Benchmark Contamination
• Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap
• Identifying and Answering Questions with False Assumptions: An Interpretable Approach
• A Question Answering Dataset for Temporal-Sensitive Retrieval-Augmented Generation
• MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents
• Never compromise with vulnerabilities: a comprehensive survey on AI governance
• TiMoE: Time-Aware Mixture of Language Experts
• Beyond Prompt-Induced Lies: Investigating LLM Deception on Benign Prompts
• Harnessing Temporal Databases for Systematic Evaluation of Factual Time-Sensitive Question-Answering in Large Language Models
• FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factuality
• AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
• A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models
• EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledge
• FoGE: Fock Space inspired encoding for graph prompting
• Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models
• TermSight: Making Service Contracts Approachable
• The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs
• KramaBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes
• LLM-Independent Adaptive RAG: Let the Question Speak for Itself
• RARE: Retrieval-Aware Robustness Evaluation for Retrieval-Augmented Generation Systems
• SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models
• Measuring Faithfulness and Abstention: An Automated Pipeline for Evaluating LLM-Generated 3-ply Case-Based Legal Arguments
• Will It Still Be True Tomorrow? Multilingual Evergreen Question Classification to Improve Trustworthy QA
• It's High Time: A Survey of Temporal Question Answering
• From Reasoning to Generalization: Knowledge-Augmented LLMs for ARC Benchmark
• VeriFastScore: Speeding up long-form factuality evaluation
• DailyQA: A Benchmark to Evaluate Web Retrieval Augmented LLMs Based on Capturing Real-World Changes
• Augmenting LLM Reasoning with Dynamic Notes Writing for Complex QA
• InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generation
• UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking
• MedBrowseComp: Benchmarking Medical Deep Research and Computer Use
• TIME: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenarios
• XRAG: Cross-lingual Retrieval-Augmented Generation
• Query-driven Document-level Scientific Evidence Extraction from Biomedical Studies
• Agentic Abstention: Do Agents Know When to Stop Instead of Act?
• Decoupling Search from Reasoning: A Vendor-Agnostic Grounding Architecture for LLM Agents
• ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?
• LLM Agents in Law: Taxonomy, Applications, and Challenges
• How Far Are LLMs from Professional Poker Players? Revisiting Game-Theoretic Reasoning with Agentic Tool Use
• LLM Ethics Benchmark: A Three-Dimensional Assessment System for Evaluating Moral Reasoning in Large Language Models
• Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications
• Information Retrieval in the Age of Generative AI: The RGB Model
• BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese
• HalluLens: LLM Hallucination Benchmark
• When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agents
• RAG-Stack: Co-Optimizing RAG Serving Performance and Quality
• MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluation
• Synergizing RAG and Reasoning: A Systematic Review
• Inducing Vulnerable Code Generation in LLM Coding Assistants
• Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
• Analyzing LLMs' Knowledge Boundary Cognition Across Languages Through the Lens of Internal Representations
• FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
• ToolRL: Reward is All Tool Learning Needs
• Memorization vs. Reasoning: Updating LLMs with New Knowledge
• Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositions
• Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory
• TALE: A Tool-Augmented Framework for Reference-Free Evaluation of Large Language Models
• Thinking Out Loud: Do Reasoning Models Know When They're Right?

Discussions

• FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation https://arxiv.org/abs/2310.03214v1 (A new paper by Google found that Perplexity and GPT-4 w/ prompting outperformed Google S [bsky, 9 points, 1 comments]
• "we present FreshPrompt, a simple few-shot prompting method that substantially boosts the performance of an LLM on FreshQA by incorporating relevant and up-to-date information retrieved from a search [bsky, 2 points, 0 comments]
• 📝 FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation 📚 "Benchmarks a diverse array of both closed and open-source large language models under a two-mode evaluation procedure [mastodon, 0 points, 0 comments]

Related

• Evaluating Open-Domain Question Answering in the Era of Large Language Models
• LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge
• Search Engines in an AI Era: The False Promise of Factual and Verifiable Source-Cited Responses
• Internet-augmented language models through few-shot prompting for open-domain question answering
• Winning Big with Small Models: Knowledge Distillation vs. Self-Training for Reducing Hallucination in QA Agents