Dataset Artefacts are the Hidden Drivers of the Declining Disruptiveness in Science
2024/02/07 by Vincent Holst, Holst, Vincent, Andres Algaba +7 · 11 voices · 3 citations
Medicine · Business, Management and Accounting · #Artificial Intelligence in Healthcare and Education #Big Data and Business Intelligence
paper · pdf · doi:10.48550/arxiv.2402.14583
Abstract
Park et al. [1] reported a decline in the disruptiveness of scientific and technological knowledge over time. Their main finding is based on the computation of CD indices, a measure of disruption in citation networks [2], across almost 45 million papers and 3.9 million patents. Due to a factual plotting mistake, database entries with zero references were omitted in the CD index distributions, hiding a large number of outliers with a maximum CD index of one, while keeping them in the analysis [1]. Our reanalysis shows that the reported decline in disruptiveness can be attributed to a relative decline of these database entries with zero references. Notably, this was not caught by the robustness checks included in the manuscript. The regression adjustment fails to control for the hidden outliers as they correspond to a discontinuity in the CD index. Proper evaluation of the Monte-Carlo simulations reveals that, because of the preservation of the hidden outliers, even random citation behaviour replicates the observed decline in disruptiveness. Finally, while these papers and patents with supposedly zero references are the hidden drivers of the reported decline, their source documents predominantly do make references, exposing them as pure dataset artefacts.
Cited by
Discussions
- Seaborn bug responsible for finding of declining disruptiveness in science [hn, 82 points, 72 comments]
- Is disruption in science decreasing? Maybe not... "Due to a factual plotting mistake, database entries with zero references were omitted in the CD index distributions, hiding a large number of outlier [bsky, 51 points, 5 comments]
- Vincent Ginis has shared a pretty compelling demonstration (at least I think so) that the original result is basically an artifact; you can replicate the "decline" with random citation behavior arxiv. [bsky, 12 points, 2 comments]
- Careful about that “declining disruptiveness” bit. Looks like it was a Seaborn error. arxiv.org/abs/2402.14583 [bsky, 9 points, 2 comments]
- Rare sighting of letter-values plots in the wild. Nicely described in the caption as "plots which first identify the median, then extend boxes outward, each covering half of the remaining data." n=2.9 [bsky, 8 points, 0 comments]
- I love this so much. Grand, implicitly political finding of meta-study turns out to be the effect of a bug in a visualization library which, in turn, is the product of floating point errors. "There is [bsky, 6 points, 1 comments]
- Via a work colleague who noted some of the error comes from a bug in a Python library (Seaborn 0.11.2) arxiv.org/pdf/2402.145... ... the decline of "disruptiveness" reported in a Nature paper is due t [bsky, 3 points, 0 comments]
- Looking at the papers citing it, I also see this working paper claiming that the entire effect can be explained by pure dataset artifacts -- papers that the dataset incorrectly recorded as having zero [bsky, 3 points, 0 comments]
- IMO you should cite every package (+ version no.) whenever possible. See here for a recent high profile case where a bug (silently dropping outliers in plots but not analyses) in a specific version of [bsky, 2 points, 1 comments]
- Seaborn bug responsible for finding of declining disruptiveness in science (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
- Oh dear, another high profile shock result turns out to be flawed. “Dataset Artefacts are the Hidden Drivers of the Declining Disruptiveness in Science” arxiv.org/abs/2402.14583 [bsky, 0 points, 0 comments]
Related