ABSTRACT

Previous research indicated that highly cited papers’ interdisciplinary patterns exhibit higher variety, lower balance and lower disparity. This paper challenges those finding, and compares the variety, balance and disparity of all papers published in 2000, categorized into six percentile-rank classes of citation rates. Our results demonstrate that, in most disciplines, highly cited papers exhibit higher variety and disparity than less cited papers, while for balance, different indicators (Gini and Shannon) lead to different results. For instance, the normalized Shannon diversity index shows that highly cited papers in social sciences and humanities have higher balance, but not those in natural sciences and engineering, while the Gini coefficient provides evidence of lower balance for highly cited papers in most specialties of those two domains. On the whole, our analysis demonstrates that highly cited papers generally exhibit higher variety, lower balance, and higher disparity than less cited papers.

Keywords:

Interdisciplinarity, Highly Cited Papers, Diversity, Indicators

1. Introduction

Over the last decades, the relationship between interdisciplinarity and scholarly impact has led a large body of literature, often obtaining conflicting results. While some studies show that interdisciplinary research leads, on average, to higher scientific impact [1] [2] [3] [4], others have obtained more nuanced results [5] [6] [7] [8] [9]. While several issues might explain these differences—such as the use of different disciplinary classifications or the focus on a different set of disciplines—a key factor is the definition and indicator of interdisciplinarity used. For instance, interdisciplinarity is considered to comprise three facets: variety, balance and disparity [10] [11]; and most research on the relationship between interdisciplinarity and scholarly impact has focused on a subset of these facets, leading to a partial or distorted view of the phenomenon. For instance, the facet of interdisciplinarity used by Lariviere and Gingras [6] is balance; their study indicates that there is no obvious relationship between interdisciplinarity and scientific impact, while moderate interdisciplinarity may still result in a higher impact.

In this context, two recent studies compared and analyzed the specific effects of variety, balance and disparity on scientific impact. Wang, Thijs [12] found that variety and disparity have a positive effect on long-term citations received by interdisciplinary research, while balance has a negative effect on its long-term citations. However, when scientific impact is assessed using short-term citations, neither variety nor disparity has a positive effect, and balance’s effect is no longer significant. The study conducted by Yegros-Yegros, Rafols [13] also shows that variety has a positive effect on citation rates, while balance and disparity have a negative effect. Yegros-Yegros, Rafols [13] indicated that highly cited papers tend to cite various disciplinary categories (higher variety), but cite little outside their disciplinary vicinity (lower disparity) and still in small proportions (lower balance).

Along these lines, several studies indicate that disparity has a positive effect on citations [2] [12] [14], and conclude that highly cited papers may exhibit higher disparity rather than lower disparity as claimed by Yegros-Yegros et al. As for balance, Wang, Thijs [12] and Yegros-Yegros, Rafols [13] both inferred that it has a negative effect on citations, while Lariviere and Gingras [6] indicate that there is no clear correlation between balance and scientific impact. As these studies are generally based on similar bibliometric datasets, it is essential to explore why these studies come to different conclusions, and understand the contributing factors involved. For instance, these conflicting conclusions are likely the result of different indicators for balance and disparity. The goal of this paper is to contribute to a better understanding of the relationship between interdisciplinarity and scientific impact, with a focus on the specific effects of variety, balance and disparity. More specifically, we investigate whether highly cited papers exhibit higher variety, lower balance and lower disparity.

We focus here on highly cited papers, which can be considered as core contributions to the advancement of knowledge of a discipline. Given that importance, highly-cited papers have been the focus of several analyses. For instance, research on science mapping has often used highly cited papers to understand and map the changing nature of science [15] [16] [17] [18]. Furthermore, given their skewed nature, bibliometric indicators based on average citation rates have become questionable in recent years [19] [20] [21], and percentile-based indicators, such as top 1% and top 10% most cited papers, have been used increasingly in research evaluation [22] [23]. Some authors even proposed that the percentile indicator should be used instead of average citation rate [24] [25] [26].

2. Methods

2.1. Data

That dataset used here comprises all journal articles published in 2000 (N = 751,766 papers) and indexed in Clarivate Analytics’ Web of Science (WoS), and the list of references to other papers indexed by the WoS (source items). The year 2000 was chosen since it was considered to be old enough to measure the (relatively) long-term scholarly impact of papers. The disciplinary classification of journals used in this study was sourced from the U.S. National Science Foundation (NSF) which categorizes each journal into one discipline and specialty (i.e., sub-discipline). This classification includes 14 general disciplines that are further refined into 143 specialties. Our study explores variety, balance and disparity from the perspective of specialties, which can be considered to correspond to Rinia’s “small interdisciplinarity” [27]. When calculating disparity, we constructed a specialty-to-specialty co-citation matrix by using five recent years of specialty-to-specialty citation relations. The co-citation matrix was also used to quantify the similar relations between specialties. The raw co-citation between specialties was normalized by cosine correlation.

2.2. Percentile-Rank Classes

Percentile-rank classes (PR) are often used to evaluate research and have been established in bibliometrics as a significant alternative to mean based indicators [25] [28]. Throughout this study, publications are categorized into percentile-rank classes in terms of their specialty in the dataset. The method used to calculate the percentile of each publication presented by Bornmann, Leydesdorff [23] was used in this study. We used the six percentile-rank classes applied by the US National Science Foundation (NSF) in their Science and Engineering Indicators [29]:

1) PR_<50th (papers with a percentile smaller than the 50th percentile);

2) PR_50th (papers with a percentile ≥ 50th and < 75th);

3) PR_75th (papers with a percentile ≥ 75th and < 90th);

4) PR_90th (papers with a percentile ≥ 90th and < 95th);

5) PR_95th (papers with a percentile ≥ 95th and < 99th);

6) PR_99th (papers with a percentile equal to or larger than the 99th percentile).

Papers categorized in PR_99th, PR_95th or PR_90th are generally considered as highly cited papers. The percentile-rank classes which include highly cited papers are defined as highly cited papers set. In this study, we calculated the interdisciplinarity of each paper in terms of three aspects, namely variety, balance and disparity, which were averaged for all papers of each percentile-rank class and specialty. The next step was to check whether the percentile-rank classes that include highly cited papers do indeed exhibit higher variety, lower balance and lower disparity.

2.3. Variety, Balance and Disparity

Interdisciplinarity is frequently measured in terms of the “diversity” of specific research areas found in the cited references of articles. The concept of diversity is used in many scientific fields, notably ecology and economics, and often consists of three aspects; variety, balance and disparity. In interdisciplinary research, variety generally refers to the number of disciplines or specialties, while balance is the evenness of the distribution of disciplines. Disparity measures the extent of which these disciplines are different from a cognitive point of view. The formulas used to measure variety, balance and disparity, along with their respective description, are listed in Table 1.

To measure balance, Yegros-Yegros, Rafols [13] used normalized Shannon diversity index while Wang, Thijs [12] used the reverse Gini coefficient. In order to thoroughly analyze the balance aspect of highly cited papers, the two methods are used in this study for comparison purposes.

3. Results

3.1. Variety

Figure 1 presents the variety distribution of the six percentile-rank classes for the 143 NSF specialties. In order to analyze the variety distribution appropriately and make the figure clearer, all specialties within the same discipline are grouped together on the horizontal axis, and the horizontal axis is labeled with the NSF discipline abbreviation (listed in Appendix 1). It shows that for most specialties, highly cited papers exhibit higher variety when compared to other papers. Only

a few specialties in Biology (such as Miscellaneous Biology) and Health (Speech-Language Pathology and Audiology) present lower variety in their highly cited papers set. These results indicate that highly cited papers generally exhibit higher variety than papers that are less cited.

3.2. Balance

Two indicators are used to assess balance: normalized Shannon diversity and reverse Gini coefficient. Figure 2 presents the normalized Shannon diversity distribution of the six percentile-rank classes for the 143 NSF specialties. It shows that in most specialties of Natural Sciences and Engineering, highly cited papers do not have lower balance than less cited papers. More specifically, only a few specialties in Biomedical Research and Biology show a visible lower balance in their highly cited papers set, while some specialties in Engineering and Technology, and Mathematics even show higher balance in their highly cited papers set. Within Social Sciences and Humanities, the majority of specialties show slightly higher balance in highly cited papers set, except in some specialties of Psychology. The results mean that from the perspective of normalized Shannon diversity, most specialties in Natural Sciences and Engineering generally do not present evidence of lower balance in their highly cited papers set. However, most specialties in Social Sciences and Humanities do present evidence of higher balance in highly cited papers set.

Figure 3 presents the reverse Gini coefficient distribution of the six percentile-rank classes for the 143 NSF specialties. For the majority of specialties (both within Natural Sciences and Engineering, and Social Sciences and Humanities) we see evidence of highly cited papers having lower balance in comparison to other papers. Only a few specialties in Biology and Social Sciences show higher balance in their highly cited papers set. These results contrast with those obtained using the normalized Shannon diversity. Normalized Shannon diversity and reverse Gini coefficient are often used to measure balance within interdisciplinary research. For instance, Yegros-Yegros, Rafols [13] and Wang, Thijs [12] analyzed the relationship between balance and citation impact using both normalized Shannon diversity and reverse Gini coefficient respectively, and the two

studies suggest that balance does have a negative effect on citation impact. The results from our study are consistent with Wang, Thijs [12], but not with that of Yegros-Yegros, Rafols [13], who limited themselves to publications within four specialties, which likely explains the differences observed (this issue is further discussed later in the paper).

3.3. Disparity

Figure 4 presents the disparity distribution of the six percentile-rank classes for the 143 NSF specialties. most specialties of Natural Sciences and Engineering show higher disparity in their highly cited papers set, while only a few of them (in Biology and Biomedical Research) show lower disparity in their highly cited papers set. Within Social Sciences and Humanities, almost all specialties show higher disparity in their highly cited papers set. This result demonstrates that highly cited papers exhibit higher disparity.

4. Discussion and Conclusion

Our study demonstrates that, in most specialties, highly cited papers exhibit higher variety and disparity in the specialties of the references they cite than less

cited papers. Results of the balance dimension of highly cited papers vary depending on the indicator used. When the balance dimension is measured using the normalized Shannon diversity index, highly cited papers in the majority of specialties within Natural Sciences and Engineering generally did not show lower balance, while in most specialties in Social Sciences and Humanities, they did show higher balance. However, the reverse Gini coefficient provides contrasting results: highly cited papers of most specialties show lower balance, regardless of whether they are in Natural Sciences and Humanities or in Social Sciences and Humanities.

One of the goals of our paper was to compare our results with those of Yegros-Yegros, Rafols [13], and to better understand discrepancies observed. Indeed, several factors can explain the difference between our results and theirs. Firstly, Yegros-Yegros, Rafols [13] did not focus on all disciplines but rather, limited their analysis to four WoS categories: Cell Biology (CBIOL), Engineering, Electrical & Electronic (EEE), Food Science and Technology (FSTA), Physics, Atomic, Molecular & Chemical (Physics-AMC). Although the NSF discipline classification system is different from WoS categories, the four WoS categories can be roughly mapped to specialties found the Biomedical Research, Engineering, Biology and Physics categories from NSF. Figure 2 does show that, in some specialties in these four disciplines, highly cited papers exhibit lower balance. Along these lines, some specialties in Biology and Biomedical Research also show lower disparity in their highly cited papers set (Figure 4). As there are some specialties in these disciplines which have a lower balance or lower disparity in their highly cited papers set, it is reasonable to infer that the specific WoS categories used by Yegros-Yegros, Rafols [13] influenced their results, and that patterns found in those specialties might not be representative of trends observed globally for the discipline. Another factor might be the length of the citation window used: Wang, Thijs [12] provided evidence that there is a citation delay in interdisciplinary research—i.e., citations to interdisciplinary research take more time to accumulate. As Yegros-Yegros, Rafols [13] used a five-year window to analyze the scientific impact of interdisciplinary research, it is likely that the interdisciplinarity papers analyzed had not, yet, reached their full potential.

Since citation would be affected by some other factors, except interdisciplinarity, such as the number of authors, institutions and countries. Some studies consider the effect of these factors and take them as control variable. We compare the studies of Wang, Thijs [12] and Yegros-Yegros, Rafols [13], with the same concept framework, similar control variable and regression analysis. The two studies take different indicators for variety and balance and the disparity measure is same. Wang, Thijs [12] make two citation period, namely long-term citation (13 years) and short-term citation (3 years). Yegros-Yegros, Rafols [13] used 5 years citations. In addition, the two studies both analyze the interdisciplinarity by Rao-Stirling diversity, Wang, Thijs [12] show that Rao-Stirling has a positive effect on citation impact and Yegros-Yegros, Rafols [13] indicate that Rao-Stirling has no significant results on citation impact. By comparing the conclusion about the disparity effect on citation impact, we can speculate the effect of control variables on the relationship between interdisciplinarity and scientific impact. Wang, Thijs [6] show that disparity has a postive effect on scientific impact for long-term citations and has a negative effect on scientific impact for short-term impact. Yegros-Yegros, Rafols [7] also show that disparity has a negative effect on scientific impact. so we think the control variable probaly make the effect of interdisciplinarity on scientific impact more accurate, but it probably cannot subvert the effect of interdisciplinarity on scientific impact. The citation period may be the reason of the univocal results about the relationship between interdisciplinarity and scientific impact. For this reason, the control variable do not considered in our study.

With regards to the effect of disparity on scientific impact, the majority of studies provide coherent conclusions [2] [12] [14] except that of Yegros-Yegros, Rafols [13], meaning that there is a good probability that highly cited papers exhibit higher disparity. However, the relationship between balance and scientific impact remains uncertain. Two measurements of balance, namely normalized Shannon diversity index and reverse Gini coefficient, yield contradicting results. When comparing both indicators, we find that there is a large difference between normalized Shannon diversity and reverse Gini coefficient when variety is high. This indicates that normalized Shannon diversity is heavily influenced by variety. According to the factor analysis performed on various diversity index by Wang, Thijs [12], reverse Gini coefficient is likely to be a more suitable index for balance. Then it means that highly cited papers are more likely to show lower balance. In conclusion, our analysis and research demonstrate that the references of highly cited papers commonly exhibit higher variety, lower balance and higher disparity.

Acknowledgements

The first author acknowledges support from the National Social Science Foundation of China (Grant 18BTQ080).

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper.

Cite this paper

Chen, S.J. and Larivière, V. (2020) Interdisciplinary Patterns of Highly Cited Papers: An Analysis from Disciplinary Perspective. Data Science and Informetrics, 1, 73-83. https://doi.org/10.4236/dsi.2020.11004

References

Appendix 1. Abbreviation of NSF Disciplines