2025/03/21 by Kocaman, Veysel, Santas, Muhammed, Gul, Yigit +2
#Computation and Language (cs.CL) #Cryptography and Security (cs.CR) #F.2.2 #FOS: Computer and information sciences #H.3 #I.2.7 #Information Retrieval (cs.IR) #Machine Learning (cs.LG)
paper · doi:10.48550/arxiv.2503.20794
We evaluate the performance of four leading solutions for de-identification of unstructured medical text - Azure Health Data Services, AWS Comprehend Medical, OpenAI GPT-4o, and John Snow Labs - on a ground truth dataset of 48 clinical documents annotated by medical experts. The analysis, conducted at both entity-level and token-level, suggests that John Snow Labs' Medical Language Models solution achieves the highest accuracy, with a 96% F1-score in protected health information (PHI) detection, outperforming Azure (91%), AWS (83%), and GPT-4o (79%). John Snow Labs is not only the only solution which achieves regulatory-grade accuracy (surpassing that of human experts) but is also the most cost-effective solution: It is over 80% cheaper compared to Azure and GPT-4o, and is the only solution not priced by token. Its fixed-cost local deployment model avoids the escalating per-request fees of cloud-based services, making it a scalable and economical choice.