Halo: Estimation and Reduction of Hallucinations in Open-Source Weak Large Language Models

2023/08/22 by Mohamed Elaraby, Mengyin Lu, Elaraby, Mohamed +12 · 3 citations
Computer Science · #Topic Modeling #Text Readability and Simplification #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2308.11764

Abstract

Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP). Although convenient for research and practical applications, open-source LLMs with fewer parameters often suffer from severe hallucinations compared to their larger counterparts. This paper focuses on measuring and reducing hallucinations in BLOOM 7B, a representative of such weaker open-source LLMs that are publicly available for research and commercial applications. We introduce HaloCheck, a lightweight BlackBox knowledge-free framework designed to quantify the severity of hallucinations in LLMs. Additionally, we explore techniques like knowledge injection and teacher-student approaches to alleviate hallucinations in low-parameter LLMs. Our experiments effectively demonstrate the reduction of hallucinations in challenging domains for these LLMs.

Cited by

• Large Language Models Hallucination: A Comprehensive Survey
• Sample, Align, Synthesize: Graph-Based Response Synthesis with ConGrs
• Rote Learning Considered Useful: Generalizing over Memorized Data in LLMs

Related

• HALoGEN: Fantastic LLM Hallucinations and Where to Find Them
• The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models
• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
• Investigating the Role of Prompting and External Tools in Hallucination Rates of Large Language Models
• A comprehensive taxonomy of hallucinations in Large Language Models