2022/07/28 by Joseph E. Serafy · 1 voice
Decision Sciences · #Academic Publishing and Open Access #scientometrics and bibliometrics research
paper · pdf · doi:10.1093/reseval/rvac024
openalex publication_date 2022/07/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29
A journal’s impact factor (JIF) for a given year is the number times its articles have been cited in the previous 2 years divided by the number of articles it published. First popularized as a tool to help librarians make decisions about their journal subscriptions (Garfield 1955), it has been improperly re-purposed by many to serve as an index of research and researcher quality. While the theme of the following tale/commentary is not new, it bears repeating, especially for the sake of new supervisors eager to simplify the difficult tasks of research and researcher performance assessment. Once upon a time, within a castle far, far away, a small group of researchers pursued truth by dutifully collecting, analyzing and reporting data—as they should. And it came to pass that a new overseer was installed who was charged with evaluating the performance of each group member by the products that they produced every year. Proclaimed was the following decree: (a) at least one peer-reviewed article must be generated per year; (b) any such article must have a journal impact factor (JIF) equal to or above that of an identified journal; and (c) for journal publications with a JIF less than said journal, multiple publications are required where their summed JIF values exceed said journal. The widespread misuse of JIFs for evaluating the quality of an individual paper and/or in decisions that affect an individual’s scientific career has received considerable attention over the last decade. The volume of articles decrying improper use of JIFs is staggering, especially since the San Francisco Declaration on Research Assessment (DORA, https://sfdora.org), which emerged from a gathering of journal editors and publishers at the 2012 annual meeting of the American Society of Cell Biology. Articles, editorials, and commentaries in high profile (= high JIF) journals such as Science (Simons 2008; Alberts 2013) and Nature (Adam 2002; Hicks 2015) indicate broad consensus that JIFs used wrongly can damage careers, journals, institutions, and the scientific enterprise itself. Some disagree, of course. JIF inventor, Eugene Garfield did not discourage its use in judging an article’s quality, yet conceded there were ‘dangers’ associated with such a practice (Garfield 2006). However, his is a minority opinion—most recent articles about JIF misuse echo Seglen (1997) who published a comprehensive (yet concise) critique of JIFs in the British Journal of Medicine. His article entitled ‘Why the impact factor of journals should not be used for evaluating research’ provides a hefty list of reasons for research and researcher evaluators to resist the temptation. Yet despite article after article beseeching those performing the evaluations of individual researchers not consider JIFs, the ‘JIF mania’ problem (Casadevall and Fang 2014) persists. And the researchers sighed upon learning of the decree. That they must meet an annual article quota did not sit well as they knew ‘salami science’ (Hoit 2007; Wawer 2019) among other sins (e.g. Cronin 2001) would invite shame upon them. More concerning was the thought that the value of the truths that they toiled to deliver would be judged not by their substance, but by a ratio born of the popularity of truths reported by others in the recent past. They felt sadness for the singled-out journal proclaimed to serve as the minimum threshold and they shook their heads at the odd calculus of summing JIFs to rise above that benchmark. When the researchers appealed to higher-ups, the latter shrugged. How else might we judge?, one said. You tilt at windmills, said another. Used wrongly, a JIF reduces a scientific contribution, replete with background, hypotheses, technique, analysis, predictions, and recommendations, into a single, ‘spreadsheet-able’ value derived from the historical reputation of its journal destination. Just as a job candidate’s skills, experience, and potential are often overshadowed by the reputation of the educational institution(s) they attended, JIFs have become entrenched symbols of scientific status. The JIF is just too convenient a measure for our increasingly burdened research (and researcher) assessors not to use as a shortcut to reading research articles and exercising their own judgment (Seglen 1997; Hicks 2015). To be fair, JIF misuse persists partly because some research evaluators lack training in how to judge the quality of research products and the performance of the scientists that report to them. Mired in administrative and clerical tasks, many evaluators have limited time to read articles fully and reflect on their importance. Many lack understanding of how bibliometrics are computed, how citation rates differ among fields, have never heard of DORA or the Leiden Manifesto, and are unaware of the amount and richness of literature dedicated to the JIF debate (e.g. Ethics in Science and Environmental Politics 2008, Volume 8, Issue 1; Scientometrics 2012, Volume 92, Issue 2). The net result is superficiality in the assessment of research products and of those that produce them. Researchers themselves, seeking prestige by proximity (Casadevall and Fang 2014) are compelled to perpetuate the practice of judging books (papers, people, and performance) by their covers. But do not despair, dear readers. Shortly after the researchers conveyed their concerns, the overseer relented and agreed to cast all mention of JIFs asunder. However, the researchers’ rejoice was subdued. The JIF monster had only retreated from view. Might it just operate unhindered from its hidden lair? The researchers returned to work, but they sharpened their truth slicing knives because the article quota beast was still striding tall and in plain sight. Then an idea struck: If served upon a fine silver platter, could this very tale satisfy the quota beast’s hunger, but also give it indigestion? Perhaps it’s the stuff of fairytales, but imagine a world where evaluators of research and researchers operated with the following R’s in mind: Read and reflect more. Recognize quantity can be the enemy of quality. Resist judgment by journal reputation ratio. Refrain from JIFrithmetic. In other words, exercise due diligence. Base assessment of scientific articles on their content, not their (journal) postal codes, whether summed, multiplied, or otherwise. Any resemblance of the people and places depicted herein is purely coincidental. The views and opinions expressed or implied in this article are those of the author and do not necessarily reflect the position of the National Marine Fisheries Service, NOAA. Conflict of interest statement. None declared.