2025/12/24 by Luo, An, Du, Jin, Tian, Fangqiao +9
#62-07 #62-08 #68T01 #68T05 #68T07 #68T50 #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #G.3 #H.2.8 #I.2.0 #I.2.6 #I.2.7 #I.5.1 #I.5.4 #Machine Learning (cs.LG) #Methodology (stat.ME)
paper · doi:10.48550/arxiv.2512.20959
Data science plays a critical role in transforming complex data into actionable insights across numerous domains. Recent developments in large language models (LLMs) have significantly automated data science workflows, but a fundamental question persists: Can these agentic AI systems truly match the performance of human data scientists who routinely leverage domain-specific knowledge? We explore this question by designing a prediction task where a crucial latent variable is hidden in relevant image data instead of tabular features. As a result, agentic AI that generates generic codes for modeling tabular data cannot perform well, while human experts could identify the important hidden variable using domain knowledge. We demonstrate this idea with a synthetic dataset for property insurance. Our experiments show that agentic AI that relies on generic analytics workflow falls short of methods that use domain-specific insights. This highlights a key limitation of the current agentic AI for data science and underscores the need for future research to develop agentic AI systems that can better recognize and incorporate domain knowledge.