Potemkin Understanding in Large Language Models

2025/06/26 by Marina Mancoridis, Bec Weeks, Mancoridis, Marina +5 · 28 voices · 7 citations
Computer Science · Social Sciences · #Computational and Text Analysis Methods #Explainable Artificial Intelligence (XAI) #Topic Modeling #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2506.21521

openalex publication_date 2025/06/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) are regularly evaluated using benchmark datasets. But what justifies making inferences about an LLM's capabilities based on its answers to a curated set of questions? This paper first introduces a formal framework to address this question. The key is to note that the benchmarks used to test LLMs -- such as AP exams -- are also those used to test people. However, this raises an implication: these benchmarks are only valid tests if LLMs misunderstand concepts in ways that mirror human misunderstandings. Otherwise, success on benchmarks only demonstrates potemkin understanding: the illusion of understanding driven by answers irreconcilable with how any human would interpret a concept. We present two procedures for quantifying the existence of potemkins: one using a specially designed benchmark in three domains, the other using a general procedure that provides a lower-bound on their prevalence. We find that potemkins are ubiquitous across models, tasks, and domains. We also find that these failures reflect not just incorrect understanding, but deeper internal incoherence in concept representations.

Citations

• Evaluating the World Model Implicit in a Generative Model
• The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
• Why AI is Harder Than We Think
• Datasheets for datasets

Cited by

• Machine understanding
• The wall confronting large language models
• Interpolative Decoding: Exploring the Spectrum of Personality Traits in LLMs
• Cognitive Foundations for Reasoning and Their Manifestation in LLMs
• Do Bias Benchmarks Generalise? Evidence from Voice-based Evaluation of Gender Bias in SpeechLLMs
• What Does Your Benchmark Really Measure? A Framework for Robust Inference of AI Capabilities
• This human study did not involve human subjects: Validating LLM simulations as behavioral evidence

Discussions

• “Potemkin Understanding in Large Language Models” A detailed analysis of the incoherent application of concepts by LLMs showing how benchmarks that reliably establish domain competence in humans can b [bsky, 16 points, 3 comments]
• My LLM coding interactions taught me that their benchmark results vastly overstate their capabilities. The paper by M. Mancoridis et al. “Potemkin Understanding in LLMs” explains and formalizes my fee [bsky, 12 points, 0 comments]
• Potemkin Understanding in Large Language Models [hn, 5 points, 0 comments]
• Muy interesante este artículo sobre cómo difieren los errores de razonamiento de personas y LLMs: arxiv.org/pdf/2506.21521 [bsky, 5 points, 1 comments]
• You might find this paper interesting! It’s about detecting the gap between how we evaluate humans understanding concepts vs LLMs doing so. The places where an LLM can produce content that would pass [bsky, 5 points, 1 comments]
• If anyone’s interested, the most positive thing I got out of grading this was I found a paper on the concept of Potemkin Understanding and how it allows LLMs to fake intelligence while getting fundame [bsky, 3 points, 1 comments]
• Favorite phrase of the day "potemkin understanding" arxiv.org/pdf/2506.21521 There is nothing surprising here but it is good that folks are trying to come up with non-anthropomorphized ways for evalua [bsky, 3 points, 0 comments]
• This might be an example of the type of incoherency that LLMs tend to exhibit. It can describe a concept, but is unable to apply the concept arxiv.org/abs/2506.21521 It "knows" what a misspelling is, [bsky, 2 points, 1 comments]
• oh but chatgpt passed the bar exam yeah, that's bullshit and this is why arxiv.org/pdf/2506.21521 [bsky, 2 points, 0 comments]
• 「ABAB韻律とは何か？」と定義を問う問題には正答できるのに、「定義を満たすように空所を埋めよ」という生成問題は正答できず、にも関わらず「（誤って生成した）文は定義を満たすか？」という分類問題には正答できる。というようにLLMが一貫性のない回答をする現象をPotemkin Undestandingと名付けて問題提起している論文 arxiv.org/abs/2506.21521 面白いんだけど、単純 [bsky, 1 points, 1 comments]
• "거대언어모델(LLM)에서 포템킨/포촘킨 이해" arxiv.org/abs/2506.21521 영어도 지식도 일천하지만 대충 살펴보자면 'AI가 실제 지성을 갖고 이해하는 게 아니라 아는 척만 한다'는 논문. 포촘킨 이해란 이를 '포촘킨파사드'에 비유해서 지은 호칭. 테드 창도 AI는 실제 지능이 아니라 응용통계일 뿐이라고 말했는데, 그와 일맥상통하는 듯. [bsky, 1 points, 0 comments]
• arxiv.org/abs/2506.21521 [bsky, 1 points, 0 comments]
• The next "AI is Dumb" paper has dropped! While it will no doubt be badly reported, this paper is really good. Good because it avoids vague words like "reasoning" and actually presents a framework for [bsky, 1 points, 1 comments]
• Is that this paper? It doesn’t mention limericks specifically, but it does mention poetic forms. arxiv.org/abs/2506.21521 [bsky, 0 points, 1 comments]
• Potemkin Understanding in Large Language Models Let's see if this term sticks: "Potemkins are to conceptual knowledge what hallucinations are to factual knowledge—hallucinations fabricate false facts; [bsky, 0 points, 0 comments]
• This is important work that can refine how evals are done. arxiv.org/abs/2506.21521 [bsky, 0 points, 0 comments]
• There’s a lot of very clear thinking in this paper! Hits close to home for me as it strongly relates to the challenge of predicting real world #robotics performance from benchmarks, or how benchmarks [bsky, 0 points, 1 comments]
• Belated follow-up: this wasn't where I first saw the point that I was trying to make, but it seems to fit into the same mode of analysis arxiv.org/abs/2506.21521 [bsky, 0 points, 0 comments]
• this excellent work shows how far from AGI large language models are, and how dangerous is their widespread use. The benchmarks used to evaluate them are wrong. #AI https://arxiv.org/abs/2506.21521 [bsky, 0 points, 0 comments]
• At the conference, Ronald Siebes discussed the limitations of LLMs. They can explain various issues well but often struggle to apply this knowledge. He emphasized the important role of symbolic AI in [bsky, 0 points, 1 comments]
• Potemkin Understanding in Large Language Models arxiv.org/abs/2506.21521 [bsky, 0 points, 0 comments]
• 言語生成AIはポチョムキン理解？物事の意味を理解していない？そりゃあそうだろうなぁとしか。でもそれを定量化するのは難しいと思っていたけれど、定量化を試みた研究があったんだねぇ。 arxiv.org/abs/2506.21521 [bsky, 0 points, 0 comments]
• the lights are on, but no one's home. - "useful work" IF trusted (verified), but this is not the path to AGI, no matter how many $T invested. "These are not the droids we are looking for" ;> Potemkin [bsky, 0 points, 0 comments]
• Score another one for the shrieking Luddites. A researcher I really respect, one of my very favorites! sez LLMs can’t think/understand like humans arxiv.org/abs/2506.21521 [bsky, 0 points, 1 comments]
• Part of the avalanche of recent papers showing incoherence of LLMs: @sendhil.bsky.social and coauthors show that LLM answers contradict their reasoning in the majority of cases, among 7 current LLM/LR [bsky, 0 points, 0 comments]
• Potemkin Understanding in Large Language Modelsていう論文おもしろい arxiv.org/abs/2506.21521 ポチョムキン理解とは「見せかけだけの、空虚な理解」。LLMがベンチマークテストなどで高いスコアを叩き出し、一見すると人間のように概念を深く理解しているように見えるが、その実態は、表面的なパターンを学習しただけで、人間のような本質的な理 [bsky, 0 points, 1 comments]
• https://bsky.app/profile/terry24777.bsky.social/post/3ltj7aqgq7s2z [bsky, 0 points, 0 comments]
• LLMのポチョムキン理解(知識の応用や、概念を一貫して使うことができない状態)に関する論文。既存の評価指数では好成績だがはりぼて状態なので、評価指数の見直しをすべきでは "[2506.21521] Potemkin Understanding in Large Language Models" https://arxiv.org/abs/2506.21521 [bsky, 0 points, 0 comments]

Related

• Line Goes Up? Inherent Limitations of Benchmarks for Evaluating Large Language Models
• BELL: Benchmarking the Explainability of Large Language Models
• Do Large Language Models Know How Much They Know?
• LLMs' Understanding of Natural Language Revealed
• MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models