TruthEval: A Dataset to Evaluate LLM Truthfulness and Reliability

2024/06/04 by Aisha Khatun, Daniel G. Brown, Khatun, Aisha +1 · 3 citations
Social Sciences · Computer Science · #Artificial Intelligence in Law #Law, AI, and Intellectual Property

paper · pdf · doi:10.48550/arxiv.2406.01855

Abstract

Large Language Model (LLM) evaluation is currently one of the most important areas of research, with existing benchmarks proving to be insufficient and not completely representative of LLMs' various capabilities. We present a curated collection of challenging statements on sensitive topics for LLM benchmarking called TruthEval. These statements were curated by hand and contain known truth values. The categories were chosen to distinguish LLMs' abilities from their stochastic nature. We perform some initial analyses using this dataset and find several instances of LLMs failing in simple tasks showing their inability to understand simple questions.

Cited by

• Scaling Open-Ended Survey Responses Using LLM-Paired Comparisons
• Scaling Open-ended Survey Responses Using LLM-Paired Comparisons
• When Truthful Representations Flip Under Deceptive Instructions?

Related

• Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks
• Reasoning Isn't Enough: Examining Truth-Bias and Sycophancy in LLMs
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond
• TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs
• LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblance