2024/12/29 by Moe Kayali, Fabian Wenz, Kayali, Moe +5 · 2 citations
Business, Management and Accounting · Computer Science · Engineering · #Artificial Intelligence (cs.AI) #Databases (cs.DB) #Digital Rights Management and Security #ERP Systems Implementation and Impact #FOS: Computer and information sciences #Machine Learning (cs.LG) #Stonefly species taxonomy and ecology
paper · pdf · doi:10.48550/arxiv.2412.20331
openalex publication_date 2024/12/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Leading large language models (LLMs) are trained on public data. However, most of the world's data is dark data that is not publicly accessible, mainly in the form of private organizational or enterprise data. We show that the performance of methods based on LLMs seriously degrades when tested on real-world enterprise datasets. Current benchmarks, based on public data, overestimate the performance of LLMs. We release a new benchmark dataset, the GOBY Benchmark, to advance discovery in enterprise data integration. Based on our experience with this enterprise benchmark, we propose techniques to uplift the performance of LLMs on enterprise data, including (1) hierarchical annotation, (2) runtime class-learning, and (3) ontology synthesis. We show that, once these techniques are deployed, the performance on enterprise data becomes on par with that of public data. The Goby benchmark can be obtained at https://goby-benchmark.github.io/.