2024/06/16 by Heidi Carolina Tamm, Anastasija Nikiforova, Tamm, Heidi Carolina +1 · 1 citation
Computer Science · Decision Sciences · #Artificial Intelligence (cs.AI) #Computational Engineering #Data Mining Algorithms and Applications #Data Quality and Management #Databases (cs.DB) #Emerging Technologies (cs.ET) #FOS: Computer and information sciences #Finance #Semantic Web and Ontologies #and Science (cs.CE)
paper · pdf · doi:10.48550/arxiv.2406.10940
openalex publication_date 2024/06/16 · openalex created_date 2024/06/19 · openalex updated_date 2026/07/28
As part of the “From Data Quality for AI to AI for Data Quality: A Systematic Review of Tools for AI-Augmented Data Quality Management in Data Warehouses” (Tamm & Nikifovora, 2025), a systematic review of DQ tools was conducted to evaluate their automation capabilities, particularly in detecting and recommending DQ rules in data warehouse - a key component of data ecosystems. To attain this objective, five key research questions were established. Q1. What is the current landscape of DQ tools? Q2. What functionalities do DQ tools offer? Q3. Which data storage systems DQ tools support? and where does the processing of the organization’s data occur? Q4. What methods do DQ tools use for rule detection? Q5. What are the advantages and disadvantages of existing solutions? Candidate DQ tools were identified through a combination of rankings from technology reviewers and academic sources. A Google search was conducted using keyword (“the best data quality tools” OR “the best data quality software” OR “top data quality tools” OR “top data quality software”) AND "2023" (search conducted in December 2023). Additionally, this list was complemented by DQ tools found in academic articles, identified with two queries in Scopus, namely "data quality tool" OR "data quality software" and ("information quality" OR "data quality") AND ("software" OR "tool" OR "application") AND "data quality rule". For selecting DQ tools for further systematic analysis, several exclusion criteria were applied. Tools from sponsored, outdated (pre-2023), non-English, or non-technical sources were excluded. Academic papers were restricted to those published within the last ten years, focusing on the computer science field. This resulted in 151 DQ tools, which are provided in the file "DQ Tools Selection". To structure the review process and facilitate answering the established questions (Q1-Q3), a review protocol was developed, consisting of three sections. The initial tool assessment was based on availability, functionality, and trialability (e.g., open-source, demo version, or free trial). Tools that were discontinued or lacked sufficient information were excluded. The second phase (and protocol section) focused on evaluating the functionalities of the identified tools. Initially, the core DQM functionalities were assessed, such as data profiling, custom DQ rule creation, anomaly detection, data cleansing, report generation, rule detection, data enrichment. Subsequently, additional data management functionalities such as master data management, data lineage, data cataloging, semantic discovery, and integration were considered. The final stage of the review examined the tools' compatibility with data warehouses and General Data Protection Regulation (GDPR) compliance. Tools that did not meet these criteria were excluded. As such, the 3rd section of the protocol evaluated the tool's environment and connectivity features, such as whether it operates in the cloud, hybrid, or on-premises, its API support, input data types (.txt, .csv, .xlsx, .json), and its ability to connect to data sources including relational and non-relational databases, data warehouses, cloud data storages, data lakes. Additionally, it assessed whether the tool processes data on-premises or in the vendor’s cloud environment. Tools were excluded based on criteria such as not supporting data warehouses or processing data externally. These protocols (filled) are available in file "DQ Tools Analysis"