Evaluating Deduplication Techniques for Economic Research Paper Titles with a Focus on Semantic Similarity using NLP and LLMs

2024/10/02 by You, Doohee, Fraiberger, S
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2410.01141

Abstract

This study investigates efficient deduplication techniques for a large NLP dataset of economic research paper titles. We explore various pairing methods alongside established distance measures (Levenshtein distance, cosine similarity) and a sBERT model for semantic evaluation. Our findings suggest a potentially low prevalence of duplicates based on the observed semantic similarity across different methods. Further exploration with a human-annotated ground truth set is completed for a more conclusive assessment. The result supports findings from the NLP, LLM based distance metrics.

Related

• A Novel Patent Similarity Measurement Methodology: Semantic Distance and Technological Distance
• Decoding Knowledge Claims: The Evaluation of Scientific Publication Contributions through Semantic Analysis
• LSH methods for data deduplication in a Wikipedia artificial dataset
• Explainable identification of similarities between entities for discovery in large text
• High precision but variable recall – comparing the performance of five deduplication tools