2025/10/31 by Xiaomeng Xu, Zahin Wahab, Xu, Xiaomeng +5
Computer Science · #Software Engineering Research #Software System Performance and Reliability #Software Testing and Debugging Techniques #cs.SE
paper · pdf · doi:10.48550/arxiv.2511.00215
22 pages. To appear in Proceedings of the ACM on Software Engineering, Volume 3, ISSTA 2026
arxiv created 2026/07/29 · arxiv updated 2026/07/31
Code-documentation inconsistencies are common and undesirable: they can lead to developer misunderstandings and software defects. This paper introduces DocPrism, a lightweight multi-language, code-documentation inconsistency detection tool. DocPrism uses a standard large language model (LLM) to analyze and explain inconsistencies, and focuses on outputting incorrectness inconsistencies. Plain use of LLMs for this task yields unacceptably high inconsistency flag rates-i.e., over 90% of functions are flagged as inconsistent with their documentation. One substantial reason is that LLMs identify natural gaps between high-level documentation and code as incompleteness inconsistencies. We introduce and apply the Local Categorization, External Filtering (LCEF) methodology: LCEF uses an LLM's local completion skills, rather than its long-term reasoning skills, to focus on reporting incorrectness inconsistencies. In our ablation study, LCEF reduces DocPrism's inconsistency flag rate from 98% to 14%, and increases F1 score from 0.22 to 0.77, compared to standard prompting techniques. On a broad evaluation across Python, TypeScript, C++, and Java, DocPrism maintains a low flag rate of 17%, and achieves a precision of 0.63 without performing any fine-tuning. We also establish a conservative lower bound across four programming languages, showing that inconsistency errors are present in 11% of code-documentation pairs. In addition, DocPrism achieves precision comparable to the state-of-the-art on an established synthetic dataset, but substantially outperforms it on our real-world Java dataset in precision (DocPrism: 0.47-0.67 vs. SOTA: 0.05-0.14).