2024/03/04 by Voraprapa Nakavachara, Nakavachara, Voraprapa, Tanapong Potipiti +3
Business, Management and Accounting · Computer Science · #AI in Service Interactions #FOS: Economics and business #FinTech, Crowdfunding, Digital Finance #General Economics (econ.GN) #Online Learning and Analytics
paper · pdf · doi:10.48550/arxiv.2403.01770
openalex publication_date 2024/03/04 · openalex created_date 2024/03/06 · openalex updated_date 2026/07/28
Generative AI technologies such as ChatGPT, Gemini, and MidJourney have made remarkable progress in recent years. Recent literature has documented ChatGPT's positive impact on productivity in areas where it has strong expertise, attributable to extensive training datasets, such as the English language and Python/SQL programming. However, there is still limited literature regarding ChatGPT's performance in areas where its capabilities could still be further enhanced. This paper aims to fill this gap. We conducted an experiment in which economics students were asked to perform writing analysis tasks in a non-English language (specifically, Thai) and math & data analysis tasks using a less frequently used programming package (specifically, Stata). The findings suggest that, on average, participants performed better using ChatGPT in terms of scores and time taken to complete the tasks. However, a detailed examination reveals that 34% of participants saw no improvement in writing analysis tasks, and 42% did not improve in math & data analysis tasks when employing ChatGPT. Further investigation indicated that higher-ability students, as proxied by their econometrics grades, were the ones who performed worse in writing analysis tasks when using ChatGPT. We also found evidence that students with better digital skills performed better with ChatGPT. This research provides insights on the impact of generative AI. Thus, stakeholders can make informed decisions to implement appropriate policy frameworks or redesign educational systems. It also highlights the critical role of human skills in addressing and complementing the limitations of technology.