High precision but variable recall – comparing the performance of five deduplication tools

2024/03/17 by Heidrun Janka, Maria‐Inti Metzendorf · 1 voice
Computer Science · #Algorithms and Data Compression

paper · pdf · doi:10.32384/jeahil20607

openalex publication_date 2024/03/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/06/26

Abstract

Deduplication methods for multiple database searches conducted for evidence syntheses differ in terms of time invested, accuracy, and comprehensiveness of identified duplicates. Deduplication tools can significantly contribute to a more efficient conduct of the search task in evidence syntheses. Widely-used tools for deduplication include reference management software (e.g. EndNote), built-in deduplication features in systematic review software (e.g. Covidence, Rayyan), and automated deduplication tools (e.g. Deduklick, SRA Deduplicator). Newer tools leverage machine learning algorithms crafted by information specialists, that encompass natural language normalization and rule-based approaches. We investigated five frequently used automated and semi-automated deduplication tools regarding their performance, core features and time efficiency in comparison to manual deduplication in EndNote using six datasets.

Discussions

• Was reading this recent article (doi.org/10.32384/jeahil20607) and noticed a couple of awesome things: 1) They used 6 datasets and included records from trial registries (clinicaltrials.gov and WHO IC [bsky, 3 points, 0 comments]

Related

• Automation of duplicate record detection for systematic reviews: Deduplicator
• LSH methods for data deduplication in a Wikipedia artificial dataset
• LSHBloom: Memory-efficient, Extreme-scale Document Deduplication
• Evaluating Deduplication Techniques for Economic Research Paper Titles with a Focus on Semantic Similarity using NLP and LLMs
• Cross-Attention Calibrated Deduplication for Retrieval-Augmented Generation System