2018/02/10 by Drahomíra Herrmannová, Drahomira Herrmannova, Robert M. Patton +3
Biochemistry, Genetics and Molecular Biology · Computer Science · Decision Sciences · Physics and Astronomy · Psychology · #Advanced Text Analysis Techniques #Audience measurement #Baseline (sea) #Bibliometrics #Biomedical Text Mining and Ontologies #Citation #Citation analysis #Citation impact #Computer science #Data science #Field (mathematics) #Intuition #Library science #MEDLINE #Political science #Psychology #Systematic review #cs.DL #physics.soc-ph #scientometrics and bibliometrics research
paper · pdf · doi:10.1007/s11192-018-2669-y
published as Herrmannova, D., Patton, R.M., Knoth, P. et al. Scientometrics (2018). https://doi.org/10.1007/s11192-018-2669-y · Accepted to journal Scientometrics
openalex publication_date 2018/02/10 · arxiv created 2018/02/13 · arxiv updated 2018/02/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
In this paper, we show that citation counts work better than a random baseline (by a margin of 10%) in distinguishing excellent research, while Mendeley reader counts don't work better than the baseline. Specifically, we study the potential of these metrics for distinguishing publications that caused a change in a research field from those that have not. The experiment has been conducted on a new dataset for bibliometric research called TrueImpactDataset. TrueImpactDataset is a collection of research publications of two types -- research papers which are considered seminal works in their area and papers which provide a literature review of a research area. We provide overview statistics of the dataset and propose to use it for validating research evaluation metrics. Using the dataset, we conduct a set of experiments to study how citation and reader counts perform in distinguishing these publication types, following the intuition that causing a change in a field signifies research contribution. We show that citation counts help in distinguishing research that strongly influenced later developments from works that predominantly discuss the current state of the art with a degree of accuracy (63%, i.e. 10% over the random baseline). In all setups, Mendeley reader counts perform worse than a random baseline.