LLMs in Coding and their Impact on the Commercial Software Engineering Landscape

2025/06/19 by Belozerov, Vladislav, Barclay, Peter J, Sami, Askhan
Computer Science · Decision Sciences · Social Sciences · #Artificial Intelligence (cs.AI) #Ethics and Social Impacts of AI #FOS: Computer and information sciences #Machine Learning (cs.LG) #Scientific Computing and Data Management #Software Engineering (cs.SE) #Software Engineering Research

paper · pdf · doi:10.48550/arxiv.2506.16653

openalex publication_date 2025/06/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large-language-model coding tools are now mainstream in software engineering. But as these same tools move human effort up the development stack, they present fresh dangers: 10% of real prompts leak private data, 42% of generated snippets hide security flaws, and the models can even ``agree'' with wrong ideas, a trait called sycophancy. We argue that firms must tag and review every AI-generated line of code, keep prompts and outputs inside private or on-premises deployments, obey emerging safety regulations, and add tests that catch sycophantic answers -- so they can gain speed without losing security and accuracy.

Citations

• Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Models
• Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications
• AI Supported Degradation of the Self Concept: A Theoretical Framework Grounded in Established Cognitive and Computational Mechanisms
• Dynamic / ME-JEPA v2.0.0-rc1: Audited World-Model Runtime and Verified Training-Corpus Artifact
• Chaos with Keywords: Exposing Large Language Models Sycophantic Hallucination to Misleading Keywords and Evaluating Defense Strategies

Related

• SOK: Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment
• Can We Trust Large Language Models Generated Code? A Framework for In-Context Learning, Security Patterns, and Code Evaluations Across Diverse LLMs
• What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants
• The Transformative Influence of LLMs on Software Development & Developer Productivity
• Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Tasks