Estimating the Self-Consistency of LLMs

2025/09/23 by Nowak, Robert
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2509.19489

Abstract

Systems often repeat the same prompt to large language models (LLMs) and aggregate responses to improve reliability. This short note analyzes an estimator of the self-consistency of LLMs and the tradeoffs it induces under a fixed compute budget B=mn, where m is the number of prompts sampled from the task distribution and n is the number of repeated LLM calls per prompt; the resulting analysis favors a rough split m,n∝√(B).

Citations

• Confidence Improves Self-Consistency in LLMs
• Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
• Self-Consistency Improves Chain of Thought Reasoning in Language Models

Related

• Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs
• Existing LLMs Are Not Self-Consistent For Simple Tasks
• Estimating LLM Consistency: A User Baseline vs Surrogate Metrics
• When is the consistent prediction likely to be a correct prediction?
• Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs