2026/04/18 by Mike Thelwall · 1 voice
Computer Science · Medicine · Social Sciences · #Artificial Intelligence in Healthcare and Education #Computational and Text Analysis Methods #Explainable Artificial Intelligence (XAI) #cs.DL #cs.IR
paper · pdf · doi:10.48550/arxiv.2605.27392
openalex publication_date 2026/04/18 · arxiv published 2026/04/18 · arxiv updated 2026/04/18 · openalex created_date 2026/05/29 · openalex updated_date 2026/07/28
Large Language Models (LLMs) like ChatGPT, DeepSeek and Gemini seem to be increasingly used for knowledge discovery, information retrieval, and knowledge summaries, including for academic topics. This can result in users being misled, such as due to hallucinations. These problems may be exacerbated for academic knowledge if LLMs base their answers on journal article abstracts when they lack full text access. To test whether the information content of abstracts can be misleading, full text articles were submitted to the GPT-OSS 120B, an LLM from OpenAI, asking it to assess separately the strength the claims for the main result in the abstract, discussion, and conclusion. Outside the social sciences and humanities, claims tended to be stronger in the abstract and conclusions than the discussion, suggesting that relying on the strength of claims in abstracts would be misleading. Thus, if LLMs ingest abstracts but not full texts, there is a risk that they will be overconfident about the findings and pass it on to users in response to relevant prompts. This is another reason to be cautious about using LLMs for academic-related knowledge discovery and summaries.