2023/07/19 by Bowen Xu, Xu, Bowen, Thanh-Dat Nguyen +19 · 1 citation
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #FOS: Computer and information sciences #Software Engineering (cs.SE) #Software Engineering Research #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2307.09765
openalex publication_date 2023/07/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Stack Overflow, the world's largest software Q&A (SQA) website, is facing a significant traffic drop due to the emergence of generative AI techniques. ChatGPT is banned by Stack Overflow after only 6 days from its release. The main reason provided by the official Stack Overflow is that the answers generated by ChatGPT are of low quality. To verify this, we conduct a comparative evaluation of human-written and ChatGPT-generated answers. Our methodology employs both automatic comparison and a manual study. Our results suggest that human-written and ChatGPT-generated answers are semantically similar, however, human-written answers outperform ChatGPT-generated ones consistently across multiple aspects, specifically by 10% on the overall score. We release the data, analysis scripts, and detailed results at https://anonymous.4open.science/r/GAI4SQA-FD5C.