2025/01/28 by Alfusainey Jallow, Jallow, Alfusainey, Sven Bugiel +1 · 1 voice
Computer Science · #Cryptography and Security (cs.CR) #Digital and Cyber Forensics #FOS: Computer and information sciences #Information and Cyber Security #Web Application Security Vulnerabilities #cs.CR
paper · pdf · doi:10.48550/arxiv.2501.16948
openalex publication_date 2025/01/28 · arxiv published 2025/01/28 · arxiv updated 2025/04/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We study the impact of Stack Overflow code evolution on the stability of prior research findings derived from Stack Overflow data and provide recommendations for future studies. We systematically reviewed papers published between 2005--2023 to identify key aspects of Stack Overflow that can affect study results, such as the language or context of code snippets. Our analysis reveals that certain aspects are non-stationary over time, which could lead to different conclusions if experiments are repeated at different times. We replicated six studies using a more recent dataset to demonstrate this risk. Our findings show that four papers produced significantly different results than the original findings, preventing the same conclusions from being drawn with a newer dataset version. Consequently, we recommend treating Stack Overflow as a time series data source to provide context for interpreting cross-sectional research conclusions.