Full-ECE: A Metric For Token-level Calibration on Large Language Models

2024/06/17 by Han Liu, Liu, Han, Yupeng Zhang +7 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2406.11345

openalex publication_date 2024/06/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Deep Neural Networks (DNNs) excel in various domains but face challenges in providing accurate uncertainty estimates, which are crucial for high-stakes applications. Large Language Models (LLMs) have recently emerged as powerful tools, demonstrating exceptional performance in language tasks. However, traditional calibration metrics such as Expected Calibration Error (ECE) and classwise-ECE (cw-ECE) are inadequate for LLMs due to their vast vocabularies, data complexity, and distributional focus. To address this, we propose a novel calibration concept called full calibration and introduce its corresponding metric, Full-ECE. Full-ECE evaluates the entire predicted probability distribution, offering a more accurate and robust measure of calibration for LLMs.

Cited by

• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs

Related

• Fill In The Gaps: Model Calibration and Generalization with Synthetic Data
• Calibration in Deep Learning: A Survey of the State-of-the-Art
• Calibration Across Layers: Understanding Calibration Evolution in LLMs
• Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
• Calibrating Large Language Models Using Their Generations Only