Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacy

2025/11/25 by Jackson, Daniel I, Jensen, Emma L, Hussain, Syed-Amad +1
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2511.19872

Abstract

Self-assessment is a key aspect of reliable intelligence, yet evaluations of large language models (LLMs) focus mainly on task accuracy. We adapted the 10-item General Self-Efficacy Scale (GSES) to elicit simulated self-assessments from ten LLMs across four conditions: no task, computational reasoning, social reasoning, and summarization. GSES responses were highly stable across repeated administrations and randomized item orders. However, models showed significantly different self-efficacy levels across conditions, with aggregate scores lower than human norms. All models achieved perfect accuracy on computational and social questions, whereas summarization performance varied widely. Self-assessment did not reliably reflect ability: several low-scoring models performed accurately, while some high-scoring models produced weaker summaries. Follow-up confidence prompts yielded modest, mostly downward revisions, suggesting mild overestimation in first-pass assessments. Qualitative analysis showed that higher self-efficacy corresponded to more assertive, anthropomorphic reasoning styles, whereas lower scores reflected cautious, de-anthropomorphized explanations. Psychometric prompting provides structured insight into LLM communication behavior but not calibrated performance estimates.

Citations

• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Pitfalls of Evaluating Language Models with Open Benchmarks
• EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code
• Evaluating LLM Metrics Through Real-World Capabilities
• Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement
• Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad
• ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning
• Free local LLMs fail rule-grounded fee computation: 0/840 exact on RuleArena airline
• Self-efficacy mechanism in human agency.
• FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs
• Internal Consistency and Self-Feedback in Large Language Models: A Survey
• LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts
• AI Benchmark Half-Life in Recursive Corpora: A Theory of Validity Decay under Semantic Leakage and Regeneration
• The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models
• PHAnToM: Persona-based Prompting Has An Effect on Theory-of-Mind Reasoning in Large Language Models
• History, Development, and Principles of Large Language Models-An Introductory Survey
• Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs
• NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
• Have Large Language Models Developed a Personality?: Applicability of Self-Assessment Tests in Measuring Personality in LLMs
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language\n Models
• Measuring Massive Multitask Language Understanding

Related

• Boosting Self-Efficacy and Performance of Large Language Models via Verbal Efficacy Stimulations
• Evaluating Large Language Models with Psychometrics
• Self-Cognition in Large Language Models: An Exploratory Study
• Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Models
• The Confidence-Competence Gap in Large Language Models: A Cognitive Study