Halo: Estimation and Reduction of Hallucinations in Open-Source Weak Large Language Models

2023/08/22 by Mohamed Elaraby, Mengyin Lu, Elaraby, Mohamed +12 · 8 citations
Computer Science · Engineering · Mathematics · Psychology · #Cognitive psychology #Computer science #Engineering #Estimation #Mathematics #Natural Language Processing Techniques #Psychology #Rank (graph theory) #Reduction (mathematics) #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2308.11764

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2023/08/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP). Although convenient for research and practical applications, open-source LLMs with fewer parameters often suffer from severe hallucinations compared to their larger counterparts. This paper focuses on measuring and reducing hallucinations in BLOOM 7B, a representative of such weaker open-source LLMs that are publicly available for research and commercial applications. We introduce HaloCheck, a lightweight BlackBox knowledge-free framework designed to quantify the severity of hallucinations in LLMs. Additionally, we explore techniques like knowledge injection and teacher-student approaches to alleviate hallucinations in low-parameter LLMs. Our experiments effectively demonstrate the reduction of hallucinations in challenging domains for these LLMs.

Cited by

• Large Language Models Hallucination: A Comprehensive Survey
• Sample, Align, Synthesize: Graph-Based Response Synthesis with ConGrs
• Rote Learning Considered Useful: Generalizing over Memorized Data in LLMs
• Hallucination Detection and Mitigation with Diffusion in Multi-Variate Time-Series Foundation Models
• Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models
• Atomic Consistency Preference Optimization for Long-Form Question Answering

Related

• HALoGEN: Fantastic LLM Hallucinations and Where to Find Them
• The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models
• Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
• Investigating the Role of Prompting and External Tools in Hallucination Rates of Large Language Models
• A comprehensive taxonomy of hallucinations in Large Language Models