Andy Wai Kan Yeung- Oral and Maxillofacial Radiology, Applied Oral Sciences, Faculty of Dentistry, The University of Hong Kong, Hong Kong, China
The relationships among various citation metrics have been probed in multiple scientific research disciplines but not neuroimaging. The aim of the current study was to assess the citation metrics of neuroimaging journals and analyze their relationships. The Journal Citation Reports (JCRs) published by Clarivate Analytics was accessed to extract relevant data for each of the 14 journals from the neuroimaging category. Pearson correlation tests were conducted to test if the citation metrics had significant correlations. Impact factor was positively correlated with citable items (r = 0.717, p = 0.004). Percentage of uncited citable items and percentage of journal self citations were partially negatively correlated with citation distribution, i.e., the percentages of citable items that contributed to 20%, 50% and 80% of total citations. The current study has implied that all the abovementioned metrics should be considered together to provide multi-faceted evaluations instead of using a single metric, at least in the neuroimaging field.
Introduction
Neuroimaging is one of the most exciting research fields as it probes into the human brain trying to understand the neurobiology of our behaviors and thoughts. According to 2018 Journal Citation Reports (JCRs) published by Clarivate Analytics, neuroimaging was ranked 29th out of 182 JCR journal categories in Science Citation Index Expanded (SCIE), sorted by aggregate impact factor. It had an aggregate impact factor of 3.996 based on a total of merely 14 journals, trailing behind neurosciences ranked 28th (aggregate impact factor of 4.015, 261 journals) but leading clinical neurology ranked 43th (aggregate impact factor of 3.498, 197 journals). It has been known that NeuroImage is the behemoth of the 14 neuroimaging journals that receives a dominating share of citations and is often co-cited with publications in neurosciences, clinical neurology and psychiatry (Yeung et al., 2017). For these tightly associated journal categories, it has been reported that the journal impact factor, the highly utilized and renowned metric for journal assessment, does not account for the percentage of uncited publications in a journal within a specific period of time. Larivière et al. (2009) has revealed that approximately 20% of publications in the medical field were uncited in around 2005, using a 2-year citation window. Even for Nobel laureates, 10% of their publications published in 2005 or before were still uncited at the end of 2010 (Egghe et al., 2011). Both van Leeuwen and Moed (2005) and Egghe (2010) have reported a negative correlation between impact factor and percentage of uncited publications, which is valid for the whole scientific field and numerous subcategories including neurosciences. It is advocated that a journal should be evaluated with multiple metrics concerning citation data, including those mentioned above as well as citation distribution data, such as the percentages of publications that contribute to 20%, 50% and 80% of total citations (Larivière et al., 2009). Little is known about the relationships among these bibliometric metrics for the neuroimaging journals, except that the field has been growing steadily for a decade since 2003 in terms of rising publication and citation counts and aggregate impact factor while maintaining a steady number of journals (Yeung et al., 2017). Therefore, the aim of the current study was to assess the neuroimaging journals and reveal if the relationships among the mentioned bibliometric metrics reported in other fields could be applied to neuroimaging category or not. The results should provide a more comprehensive view for readers to evaluate the performances of neuroimaging journals apart from using the single value of impact factor.
Materials and Methods
The JCR database hosted by Clarivate Analytics was accessed on 17 July 2018. Data was extracted from JCR Year 2017 and SCIE edition. All 14 journals indexed in neuroimaging category were analyzed in terms of:
1. Impact factor 2017, inclusive of its accountable total citations (citations in 2017 to items published in 2015 and 2016) and citable items (published in 2015 and 2016);
2. Percentage of journal self citations (i.e., abovementioned total citations contributed by the journal itself);
3. Percentage of uncited citable items; and
4. Percentage of citable items that contributed to 20%, 50% and 80% of the abovementioned total citations.
Two-tailed Pearson correlation tests were conducted in SPSS 25.0 (IBM, New York, NY, USA) to test if there existed significant association among these metrics. Tests were defined as significant if p < 0.05.
Results
The details of the citation metrics are listed in Table 1. The journals were listed in descending order of impact factor. As expected, the top-ranked NeuroImage had the largest number of total citations and citable items. Though it had the 2nd most number of uncited citable items (130), it actually had the smallest percentage of them (7.8% only). It ranked 5th in terms of percentage of journal self citations. The 2nd largest journal in terms of total citations and citable items was American Journal of Neuroradiology; it ranked 5th in terms of impact factor and 1st in terms of percentage of journal self citations.
Table 1. Various citation metrics of neuroimaging journals.
Pearson correlation tests have revealed that impact factor was positively correlated with the number of citable items (r = 0.717, p = 0.004), and negatively correlated with percentage of uncited citable items (r = −0.827, p < 0.001). A scatter plot of log-10 impact factor against percentage of uncited citable items (Supplementary Figure S1) was prepared in a similar fashion as van Leeuwen and Moed (2005; see their Figures 1, 2). If a neuroimaging journal had an impact factor approaching 1, its percentage of uncited citable items would be in the range of 40% to 50%. If it had an impact factor approaching 0.1, the percentage would be greater than 90%.
Meanwhile, percentage of uncited citable items was partially negatively correlated with citation distribution, namely with percentage of citable items that contributed to 20% and 50% of total citations (r = −0.581, p = 0.029 and r = −0.658, p = 0.010 respectively; Table 2).
Table 2. Pearson correlation coefficients between various citation parameters of neuroimaging journals.
Discussion
The current study has revealed the performance of neuroimaging journals in terms of citation metrics. Though there is limited data in the existing literature confirming or rebutting the relationship between impact factor and the number of published articles (citable items), the current results have implied that, at least for neuroimaging journals, size really matters.
Based on the data reported by van Leeuwen and Moed (2005) and Egghe (2010) has established that the impact factor of journals and their uncitedness factor (i.e., the percentage of uncited articles) follows a relationship best described with a horizontal S-shape curve. At a first glance, the Supplementary Figure S1 of the current study seemed not to demonstrate the characteristic S-shape. Upon a closer examination of the original data presented by van Leeuwen and Moed (2005), the neuroimaging journal data from the current study actually conformed to the middle segment of the S-shape, which was a relatively straight line when the impact factor range was between 0.1 and 10 (see their Figures 1, 2). This straight segment held true for the overall SCI publications and subcategories namely the biochemistry and molecular biology, inorganic chemistry, physical chemistry (van Leeuwen and Moed, 2005), as well as immunology and surgical journals (Weale et al., 2004). Our interpolations that impact factor approaching 1 and 0.1 would imply percentage of uncited citable items being 40%–50% and >90% were also consistent to their findings (van Leeuwen and Moed, 2005). Without the existence of journals with extremely large (>10) or small (<0.1) impact factor, the plot representing neuroimaging journals did not have the curved segments at the extremities and thus the Pearson correlation coefficient between these two metrics, −0.827, was larger than that of neurosciences (−0.64), general and internal medicine (−0.69) and the overall SCI category (−0.63) reported by them (van Leeuwen and Moed, 2005).
Next, the magnitude of uncitedness was compared to existing studies. In the dated literature, it was reported that there existed 0.45%–49.9% of articles that remained uncited at least for 4 years after publication in the biomedical fields ranging from chemistry to pharmacology (Stern, 1990). A recent report has revealed that the percentages of articles published in 2000 remained uncited 1 year and 2 years after publication were around 40%–80% and 20%–60% respectively, with biology articles having percentages of 40% and ~20% respectively (van Noorden, 2017). These studies have shown that the proportions of uncited articles in various research fields were largely diversified. The survey by Larivière et al. (2009) has found that the medical field publications had 20% of uncitedness in 2005 using a 2-year citation window, which has been decreasing over time. Weale et al. (2004) similarly concluded that 23.7% of articles published in immunology and surgical journals were uncited. Within the 14 journals of the neuroimaging field, the current results reported the uncitedness ranging from 7.8% to 91.2%, and an overall of 18.2%. This was somehow comparable to the findings by Larivière et al. (2009).
Meanwhile, in the medical field the percentages of articles that contributed to 20%, 50% and 80% of total citations were 2%, 11% and 33% respectively (Larivière et al., 2009). For immunology and surgical journals, the percentages of articles that contributed to 50% of total citations were 15%–18% (Weale et al., 2004). For a cardiovascular journal, 14% of articles contributed to 50% of the total citations (Opthof et al., 2004). If the percentages from each neuroimaging journal were averaged, the respective equivalent values would be 4.4%, 18.2% and 57.3%. This implied that citations to publications from neuroimaging journals were probably more dispersed than those in the medical field. From Table 2, one would realize that impact factor had no significant correlation with any of these three percentages, implying that these metrics might indeed reveal unique assessment features distinctive from the impact factor. These three percentages did have partial correlation with the percentage of journal self citations and uncited citable items (Table 2). However, the exact relationships remained to be established. One important consideration for this, and also limitation of the current analysis, is that the citation count used to calculate impact factor includes citations to publications that are not counted as citable items (Moed and Van Leeuwen, 1995, 1996). Therefore, a journal editor could boost its impact factor by publishing non-“citable items” that self cite a lot of its “citable items” (Falagas and Alexiou, 2008), which complicates the underlying behavior of citations. A study by Opthof revealed that such journal self citation practice in cardiovascular science could inflate the journal impact factor by as much as 69% (Opthof, 2013). Another limitation is the small sample size of n = 14. This study is similar to previous studies in evaluating the relationship of journal impact factor and other metrics by considering a particular JCR journal category. Because the Neuroimaging category consists of 14 journals only, the current sample size already covers the entire category. It is a common practice to use impact factor to evaluate journals, validate scientific relevance of researchers or research programs and therefore decide employment, funding and tenure (Hecht et al., 1998). Meanwhile, the San Francisco Declaration on Research Assessment (DORA1) developed in 2012 urged for the considerations of scientific content of articles rather than publication metrics for hiring, tenure and promotion decisions. Though the current study was not aimed to judge the appropriateness of such usage of the impact factor, the current results have implied that all the abovementioned journal metrics should be considered together, instead of a single metric, to provide a more comprehensive understanding of journal performance, at least in the neuroimaging field. Their extended use for evaluation purposes of individual researchers, however, may be deemed too advanced and complex. For instance, uncited articles may be “sleeping beauties,” which can be uncited for a long time, but eventually highly cited (van Raan, 2004; Yeung and Ho, 2018). It is unfeasible to evaluate researchers based on complex metrics that may not truly reflect the scientific value of their work.
Conclusions
The current study has evaluated the relationships among various citation metrics for the neuroimaging journals. Impact factor positively correlated with the number of published citable items, but negatively correlated with percentage of uncited citable items. Percentages of citable items that contributed to 20%, 50% and 80% of total citations did not correlate with impact factor. These metrics reflecting citation distribution should be considered together with other citation metrics when one is assessing neuroimaging journal performance.
Data Availability
All datasets generated for this study are included in the manuscript and the supplementary files.
Author Contributions
The author confirms being the sole contributor of this work and has approved it for publication.
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Conflict of Interest Statement
The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Footnotes
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnhum.2018.00523/full#supplementary-material
FIGURE S1 | Scatter plot of log-10 impact factor 2017 against percentage of uncited citable items.
References
Egghe, L. (2010). The distribution of the uncitedness factor and its functional relation with the impact factor. Scientometrics 83, 689–695. doi: 10.1007/s11192-009-0130-y
Egghe, L., Guns, R., and Rousseau, R. (2011). Thoughts on uncitedness: nobel laureates and fields medalists as case studies. J. Am. Soc. Inf. Sci. Technol. 62, 1637–1644. doi: 10.1002/asi.21557
Falagas, M. E., and Alexiou, V. G. (2008). The top-ten in journal impact factor manipulation. Arch. Immunol. Ther. Exp. 56, 223–226. doi: 10.1007/s00005-008-0024-5
Hecht, F., Hecht, B. K., and Sandberg, A. A. (1998). The journal “impact factor”: a misnamed, misleading, misused measure. Cancer Genet. Cytogenet. 104, 77–81. doi: 10.1016/S0165-4608(97)00459-7
Larivière, V., Gingras, Y., and Archambault, É. (2009). The decline in the concentration of citations, 1900–2007. J. Am. Soc. Inf. Sci. Technol. 60, 858–862. doi: 10.1002/asi.21011
Moed, H. F., and Van Leeuwen, T. N. (1995). Improving the accuracy of institute for scientific information’s journal impact factors. J. Am. Soc. Inf. Sci. 46, 461–467. doi: 10.1002/(sici)1097-4571(199507)46:6<461::aid-asi5>3.0.co;2-g
Moed, H. F., and Van Leeuwen, T. N. (1996). Impact factors can mislead. Nature 381, 186–186. doi: 10.1038/381186a0
Opthof, T. (2013). Inflation of impact factors by journal self-citation in cardiovascular science. Neth. Heart J. 21, 163–165. doi: 10.1007/s12471-013-0384-0
Opthof, T., Coronel, R., and Piper, H. M. (2004). Impact factors: no totum pro parte by skewness of citation. Cardiovasc. Res. 61, 201–203. doi: 10.1016/j.cardiores.2003.11.023
Stern, R. E. (1990). Uncitedness in the biomedical literature. J. Am. Soc. Inf. Sci. 41, 193–196. doi: 10.1002/(sici)1097-4571(199004)41:3<193::aid-asi5>3.0.co;2-b
van Leeuwen, T. N., and Moed, H. F. (2005). Characteristics of journal impact factors: the effects of uncitedness and citation distribution on the understanding of journal impact factors. Scientometrics 63, 357–371. doi: 10.1007/s11192-005-0217-z
van Noorden, R. (2017). The science that’s never been cited. Nature 552, 162–164. doi: 10.1038/d41586-017-08404-0
van Raan, A. F. (2004). Sleeping beauties in science. Scientometrics 59, 467–472. doi: 10.1023/B:SCIE.0000018543.82441.f1
Weale, A. R., Bailey, M., and Lear, P. A. (2004). The level of non-citation of articles within a journal as a measure of quality: a comparison to the impact factor. BMC Med. Res. Methodol. 4:14. doi: 10.1186/1471-2288-4-14
Yeung, A. W. K., and Ho, Y.-S. (2018). Identification and analysis of classic articles and sleeping beauties in neurosciences. Curr. Sci. 114, 2039–2044. doi: 10.18520/cs/v114/i10/2039-2044
Keywords: bibliometric, citation analysis, citation distribution, impact factor, neuroimaging, uncitedness
Citation: Yeung AWK (2019) Higher Impact Factor of Neuroimaging Journals Is Associated With Larger Number of Articles Published and Smaller Percentage of Uncited Articles. Front. Hum. Neurosci. 12:523. doi: 10.3389/fnhum.2018.00523
Received: 12 October 2018; Accepted: 11 December 2018;
Published: 04 January 2019.
Edited by:
Mikhail Lebedev, Duke University, United StatesReviewed by:
Tobias Opthof, Academic Medical Center (AMC), NetherlandsThed Van Leeuwen, Leiden University, Netherlands
Copyright © 2019 Yeung. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Andy Wai Kan Yeung, ndyeung@hku.hk