2024/07/18 by Chris Brown, Brown, Chris, Jason Cusati +1
Computer Science · Medicine · #AI in Service Interactions #Artificial Intelligence in Healthcare and Education #FOS: Computer and information sciences #Online Learning and Analytics #Software Engineering (cs.SE)
paper · pdf · doi:10.48550/arxiv.2407.13900
openalex publication_date 2024/07/18 · openalex created_date 2024/09/26 · openalex updated_date 2026/07/28
Background: Recent innovations in generative artificial intelligence (AI) have transformed how programmers develop and maintain software. The advanced capabilities of generative AI tools in supporting development tasks have led to a rise in their adoption within software engineering (SE) workflows. However, little is known about how AI tools perceive evidence-based practices supported by empirical SE research. Aim: To this end, we explore the "beliefs" of generative AI tools increasingly used to support software development in practice. Method: We conduct a preliminary evaluation conceptually replicating prior work to investigate 17 evidence-based claims across five generative AI tools. Results: Our findings demonstrate generative AI tools have ambiguous beliefs regarding research claims and lack credible evidence to support responses. Conclusions: Based on our results, we provide implications for practitioners integrating generative AI-based systems into development contexts and shed light on future research directions to enhance the reliability and trustworthiness of generative AI -- aiming to increase awareness and adoption of evidence-based SE research findings in practice.