2019/12/30 by Laura Kinkead, Kinkead, Laura, Ahmed Allam +3 · 1 citation
Health Professions · Social Sciences · #Computation and Language (cs.CL) #Computers and Society (cs.CY) #FOS: Computer and information sciences #Health Literacy and Information Accessibility #Health Sciences Research and Education #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Misinformation and Its Impacts
paper · pdf · doi:10.48550/arxiv.1912.12999
openalex publication_date 2019/12/30 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28
Patients increasingly turn to search engines and online content before, or in\nplace of, talking with a health professional. Low quality health information,\nwhich is common on the internet, presents risks to the patient in the form of\nmisinformation and a possibly poorer relationship with their physician. To\naddress this, the DISCERN criteria (developed at University of Oxford) are used\nto evaluate the quality of online health information. However, patients are\nunlikely to take the time to apply these criteria to the health websites they\nvisit. We built an automated implementation of the DISCERN instrument (Brief\nversion) using machine learning models. We compared the performance of a\ntraditional model (Random Forest) with that of a hierarchical encoder\nattention-based neural network (HEA) model using two language embeddings, BERT\nand BioBERT. The HEA BERT and BioBERT models achieved average F1-macro scores\nacross all criteria of 0.75 and 0.74, respectively, outperforming the Random\nForest model (average F1-macro = 0.69). Overall, the neural network based\nmodels achieved 81% and 86% average accuracy at 100% and 80% coverage,\nrespectively, compared to 94% manual rating accuracy. The attention mechanism\nimplemented in the HEA architectures not only provided 'model explainability'\nby identifying reasonable supporting sentences for the documents fulfilling the\nBrief DISCERN criteria, but also boosted F1 performance by 0.05 compared to the\nsame architecture without an attention mechanism. Our research suggests that it\nis feasible to automate online health information quality assessment, which is\nan important step towards empowering patients to become informed partners in\nthe healthcare process.\n