Quantifying the interdisciplinarity of a research is a relevant problem in the evaluative bibliometrics. The concept of interdisciplinarity is ambiguous and multidimensional. Thus, different measures of interdisciplinarity have been propose in the literature. However, few studies have proposed interdisciplinary metrics without previously defining classification sets, and no one use the co-citation network for this purpose. In this study we propose an interdisciplinary metric based on the co-citation network. This is a way to define the publication's field without resorting to pre-defined classification sets. We present a characterization of a publication's field and then we use this definition to propose a new metric of the interdisciplinarity degree for publications (papers) and journals as units of analysis. The proposed measure has an aggregative property that makes it scalable from a paper individually to a set of them (journal) without more than adding the numerators and denominators in the proportions that define this new indicator. Moreover, the aggregated value of two or more units is strictly among all the individual values.
This paper analyzes the effect of interdisciplinarity on the scientific impact of individual papers. Using all the papers published in Web of Science in 2000, we define the degree of interdisciplinarity of a given paper as the percentage of its cited references made to journals of other disciplines. We show that, although for all disciplines combined there is no clear correlation between the level of interdisciplinarity of papers and their citation rates, there are nonetheless some disciplines in which a higher level of interdisciplinarity is related to a higher citation rates. For other disci…
A citation-based indicator for interdisciplinarity has been missing hitherto among the set of available journal indicators. In this study, we investigate network indicators (betweenness centrality), journal indicators (Shannon entropy, the Gini coefficient), and more recently proposed Rao-Stirling measures for "interdisciplinarity." The latter index combines the statistics of both citation distributions of journals (vector-based) and distances in citation networks among journals (matrix-based). The effects of various normalizations are specified and measured using the matrix of 8,207 journals …
Journals were central to Eugene Garfield's research interests. Among other things, journals are considered as units of analysis for bibliographic databases such as the Web of Science (WoS) and Scopus. In addition to disciplinary classifications of journals, journal citation patterns span networks across boundaries to variable extents. Using betweenness centrality (BC) and diversity, we elaborate on the question of how to distinguish and rank journals in terms of interdisciplinarity. Interdisciplinarity, however, is difficult to operationalize in the absence of an operational definition of disc…
Many studies have sought to identify interdisciplinary research as a function of the diversity of disciplines identified in an article's references or citations. However, given the constant evolution of the scientific landscape, disciplinary boundaries are shifting and blurring, making it increasingly difficult to describe research within a strict taxonomy. In this work, we explore the potential for graph learning methods to learn embedded representations for research papers that encode their 'interdisciplinarity' in a citation network. This facilitates the identification of interdisciplinary …
Questions of definition and measurement continue to constrain a consensus on the measurement of interdisciplinarity. Using Rao-Stirling (RS) Diversity produces sometimes anomalous results. We argue that these unexpected outcomes can be related to the use of "dual-concept diversity" which combines "variety" and "balance" in the definitions (ex ante). We propose to modify RS Diversity into a new indicator (DIV) which operationalizes variety, balance, and disparity independently and then combines them ex post. "Balance" can be measured using the Gini coefficient. We apply DIV to the aggregated ci…
Using "Analyze Results" at the Web of Science, one can directly generate overlays onto global journal maps of science. The maps are based on the 10,000+ journals contained in the Journal Citation Reports (JCR) of the Science and Social Science Citation Indices (2011). The disciplinary diversity of the retrieval is measured in terms of Rao-Stirling's "quadratic entropy." Since this indicator of interdisciplinarity is normalized between zero and one, the interdisciplinarity can be compared among document sets and across years, cited or citing. The colors used for the overlays are based on Blonde…
The citation impact of Environment and Planning B can be visualized using its citation relations with journals in its environment as the links of a network. The size of the nodes is varied in correspondence to the relative citation impact in this environment. Additionally, one can correct for the effect of within-journal "self"-citations. The network can be partitioned and clustered using algorithms from social network analysis. After transposing the matrix in terms of rows and columns, the citing patterns can be mapped analogously. Citing patterns reflect the activity of the community of auth…
The tremendous advances in computer science in the last few decades have provided the platform to address and solve complex problems using interdisciplinary research. In this paper, we investigate how the extent of interdisciplinarity in computer science domain (which is further divided into 24 research fields) has changed over the last 50 years. To this end, we collect a massive bibliographic dataset with rich metadata information. We start with quantifying interdisciplinarity of a field in terms of the diversity of topics and citations. We then analyze the effect of interdisciplinary researc…
Understanding the dynamics of authors is relevant to predict and quantify performance in science. While the relationship between recent and future citation counts is well-known, many relationships between scholarly metrics at the author-level remain unknown. In this context, we performed an analysis of author-level metrics extracted from subsequent periods, focusing on visibility, productivity and interdisciplinarity. First, we investigated how metrics controlled by the authors (such as references diversity and productivity) affect their visibility and citation diversity. We also explore the r…
There is an overall perception of increased interdisciplinarity in science, but this is difficult to confirm quantitatively owing to the lack of adequate methods to evaluate subjective phenomena. This is no different from the difficulties in establishing quantitative relationships in human and social sciences. In this paper we quantified the interdisciplinarity of scientific journals and science fields by using an entropy measurement based on the diversity of the subject categories of journals citing a specific journal. The methodology consisted in building citation networks using the Journal …
Citation networks have fed numerous works in scientific evaluation, science mapping (and more recently large-scale network studies) for decades. The variety of citation behavior across scientific fields is both a research topic in sociology of science, and a problem in scientific evaluation. Normalization, tantamount to a particular weighting of links in the citation network, is necessary for allowing across-field comparisons of citation scores and interdisciplinary studies. In addition to classical normalization which drastically reduces all variability factors altogether, two tracks of resea…
A comparative study is done of interdisciplinary citations in 2013 between physics, chemistry, and molecular biology, in Brazil, South Korea, Turkey, and USA. Several surprising conclusions emerge from our tabular and graphical analysis: The cross-science citation rates are in general strikingly similar, between Brazil, South Korea, Turkey, and USA. One apparent exception is the comparatively more tenuous relation between molecular biology and physics in Brazil and USA. Other slight exceptions are the higher amount of citing of physicists by chemists in South Korea, of chemists by molecular bi…
In a context of ever more specialized scientists, interdisciplinarity receives increasing attention as innovating ideas are often situated where the disciplines meet. In many countries science policy makers installed dedicated funding programs and policies. This induces a need for specific tools for their support. There is however not yet a generally accepted quantitative method or set of criteria to recognize and evaluate interdisciplinary research outputs (Tracking and evaluating interdisciplinary research: metrics and maps, 12th ISSI Conference, 2009). Interdisciplinarity also takes on very…
Patterns of interdisciplinarity in science can be quantified through diverse complementary dimensions. This paper studies as a case study the scientific environment of a generalist journal in Geography, Cybergeo, in order to introduce a novel methodology combining citation network analysis and semantic analysis. We collect a large corpus of around 200,000 articles with their abstracts and the corresponding citation network that provides a first citation classification. Relevant keywords are extracted for each article through text-mining, allowing us to construct a semantic classification. We s…
There is demand from science funders, industry, and the public that science should become more risk-taking, more out-of-the-box, and more interdisciplinary. Is it possible to tell how interdisciplinary and out-of-the-box scientific papers are, or which papers are mainstream? Here we use the bibliographic coupling network, derived from all physics papers that were published in the Physical Review journals in the past century, to try to identify them as mainstream, out-of-the-box, or interdisciplinary. We show that the network clusters into scientific fields. The position of individual papers wi…
The growing complexity of challenges involved in scientific progress demands ever more frequent application of competencies and knowledge from different scientific fields. The present work analyzes the degree of collaboration among scientists from different disciplines in order to identify the most frequent "combinations of knowledge" in research activity. The methodology adopts an innovative bibliometric approach based on the disciplinary affiliation of publication co-authors. The field of observation includes all publications (173,134) indexed in the Science Citation Index Expanded (SCI-E) f…
Citation impact is commonly assessed using direct, first-order citation relations. We consider here instead the indirect influence of publications on new publications via citations. We present a novel method to quantify the higher-order citation influence of publications, considering both direct, or first-order, and indirect, or higher-order citations. In particular, we are interested in higher-order citation influence at the level of disciplines. We apply this method to the whole Web of Science data at the level of disciplines. We find that a significant amount of influence -- 42% -- stems fr…
Researchers' networks have been subject to active modeling and analysis. Earlier literature mostly focused on citation or co-authorship networks reconstructed from annotated scientific publication databases, which have several limitations. Recently, general-purpose web search engines have also been utilized to collect information about social networks. Here we reconstructed, using web search engines, a network representing the relatedness of researchers to their peers as well as to various research topics. Relatedness between researchers and research topics was characterized by visibility boos…
Nowadays, scientific challenges usually require approaches that cross traditional boundaries between academic disciplines, driving many researchers towards interdisciplinarity. Despite its obvious importance, there is a lack of studies on how to quantify the influence of interdisciplinarity on the research impact, posing uncertainty in a proper evaluation for hiring and funding purposes. Here we propose a method based on the analysis of bipartite interconnected multilayer networks of citations and disciplines, to assess scholars, institutions and countries interdisciplinary importance. Using d…
Interdisciplinary research increasingly fuels innovation, and is considered to be a key to tomorrow breakthrough. Yet little is known about whether interdisciplinary research manifests delayed impact. Here, we use the time to reach the citation peak to quantify the highest impact time and citation dynamics, and examine its relationship with interdisciplinarity. Using large scale publication datasets, our results suggest that interdisciplinary papers show significant delayed impact both microscopically per paper and macroscopically collectively, as it takes longer time for interdisciplinary pap…
In December 2016, after several years of development, Elsevier launched a set of transparent, comprehensive, current, and freely-available journal citation metrics called CiteScore metrics. Most of the CiteScore metrics are static, annual indicators calculated from the dynamic Scopus citation index. In the spirit of recent public statements on the responsible use of metrics, we outline the desirable characteristics of journal citation metrics, discuss how we decided on the cited and citing publications years and document types to be used for CiteScore metrics, and detail the precise method of …
The present paper takes its place in the stream of studies that analyze the effect of interdisciplinarity on the impact of research output. Unlike previous studies, in this study the interdisciplinarity of the publications is not inferred through their citing or cited references, but rather by identifying the authors' designated fields of research. For this we draw on the scientific classification of Italian academics, and their publications as indexed in the WoS over a five-year period (2004-2008). We divide the publications in three subsets on the basis the nature of co-authorship: those pap…
Evaluating and comparing the academic performance of a journal, a researcher or a single paper has long remained a critical, necessary but also controversial issue. Most of existing metrics invalidate comparison across different fields of science or even between different types of papers in the same field. This paper proposes a new metric, called return on citation (ROC), which is simply a citation ratio but applies to evaluating the paper, the journal and the researcher in a consistent way, allowing comparison across different fields of science and between different types of papers and discou…
I describe a simple modification which can be applied to any citation count-based index (e.g. Hirsch's h-index) quantifying a researcher's publication output. The key idea behind the proposed approach is that the merit for the citations of a paper should be distributed amongst its authors according to their relative contributions. In addition to producing inherently fairer metrics I show that the proposed modification has the potential to partially normalize for the unfair effects of honorary authorship and thus discourage this practice.
The prediction of the long-term impact of a scientific article is challenging task, addressed by the bibliometrician through resorting to a proxy whose reliability increases with the breadth of the citation window. In the national research assessment exercises using metrics the citation window is necessarily short, but in some cases is sufficient to advise the use of simple citations. For the Italian VQR 2011-2014, the choice was instead made to adopt a linear weighted combination of citations and journal metric percentiles, with weights differentiated by discipline and year. Given the strateg…
In this paper we present "citation success index", a metric for comparing the citation capacity of pairs of journals. Citation success index is the probability that a random paper in one journal has more citations than a random paper in another journal (50% means the two journals do equally well). Unlike the journal impact factor (IF), the citation success index depends on the broadness and the shape of citation distributions. Also, it is insensitive to sporadic highly-cited papers that skew the IF. Nevertheless, we show, based on 16,000 journals containing ~2.4 million articles, that the cita…
We generated networks of journal relationships from citation and download data, and determined journal impact rankings from these networks using a set of social network centrality metrics. The resulting journal impact rankings were compared to the ISI IF. Results indicate that, although social network metrics and ISI IF rankings deviate moderately for citation-based journal networks, they differ considerably for journal networks derived from download data. We believe the results represent a unique aspect of general journal impact that is not captured by the ISI IF. These results furthermore ra…
This study examines the correlational relationships between local journal authorship, local and external citation counts, full-text downloads, link-resolver clicks, and four global journal impact factor indices within an all-disciplines journal collection of 12,200 titles and six subject subsets at the University of Illinois at Urbana-Champaign (UIUC) Library. While earlier investigations of the relationships between usage (downloads) and citation metrics have been inconclusive, this study shows strong correlations in the all-disciplines set and most subject subsets. The normalized Eigenfactor…
ResearchGate has emerged as a popular professional network for scientists and researchers in a very short span of time. Similar to Google Scholar, the ResearchGate indexing uses an automatic crawling algorithm that extracts bibliographic data, citations and other information about scholarly articles from various sources. However, it has been observed that the two platforms often show different publication and citation data for the same institutions, journals and authors. This paper, therefore, attempts to analyse and measure the differences in publication counts, citations and different metric…
Citation count is a quantifiable measure to indicate the number of times an article is cited by other articles. It is believed that if an article is cited often then it must be an important or influential article; however, there is no guarantee that the most cited articles are good in quality. In this paper, the author suggests argumentation count, a new metric for citation analysis. The proposed metric, argumentation count is a triplet of quantities for each concept of an article that helps in providing a quantifiable measure about the usefulness of an article.
The launch of Google Scholar Citations and Google Scholar Metrics may provoke a revolution in the research evaluation field as it places within every researchers reach tools that allow bibliometric measuring. In order to alert the research community over how easily one can manipulate the data and bibliometric indicators offered by Google s products we present an experiment in which we manipulate the Google Citations profiles of a research group through the creation of false documents that cite their documents, and consequently, the journals in which they have published modifying their H index.…
The new web-based academic communication platforms do not only enable researchers to better advertise their academic outputs, making them more visible than ever before, but they also provide a wide supply of metrics to help authors better understand the impact their work is making. This study has three objectives: a) to analyse the uptake of some of the most popular platforms (Google Scholar Citations, ResearcherID, ResearchGate, Mendeley and Twitter) by a specific scientific community (bibliometrics, scientometrics, informetrics, webometrics, and altmetrics); b) to compare the metrics availab…
Despite the increasing use of citation-based metrics for research evaluation purposes, we do not know yet which metrics best deliver on their promise to gauge the significance of a scientific paper or a patent. We assess 17 network-based metrics by their ability to identify milestone papers and patents in three large citation datasets. We find that traditional information-retrieval evaluation metrics are strongly affected by the interplay between the age distribution of the milestone items and age biases of the evaluated metrics. Outcomes of these metrics are therefore not representative of th…
Citation metrics are analytic measures used to evaluate the usage, impact and dissemination of scientific research. Traditionally, citation metrics have been independently measured at each level of the publication pyramid, namely at the article-level, at the author-level, and at the journal-level. The most commonly used metrics have been focused on journal-level measurements, such as the Impact Factor and the Eigenfactor, as well as on researcher-level metrics like the Hirsch index (h-index) and i10 index. On the other hand, reliable article-level metrics are less widespread, and are often res…
The purpose of citation indexes and metrics is intended to be a measure for scientific innovation and quality for researchers, journals, and institutions. However, those metrics are often prone to abuse and manipulation by excessive and unethical self-citations induced by authors, reviewers, editors, or journals. Identifying whether there are or not legitimate reasons for self-citations is normally determined during the review process, where the participating parts may have intrinsic incentives, rendering the legitimacy of self-citations, after publication, questionable. In this paper, we cond…
The launch of Google Scholar Metrics as a tool for assessing scientific journals may be serious competition for Thomson Reuters Journal Citation Reports, and for Scopus powered Scimago Journal Rank. A review of these bibliometric journal evaluation products is performed. We compare their main characteristics from different approaches: coverage, indexing policies, search and visualization, bibliometric indicators, results analysis options, economic cost and differences in their ranking of journals. Despite its shortcomings, Google Scholar Metrics is a helpful tool for authors and editors in ide…
Computing such correlation coefficient would be straightforward had we had available the rankings given by the prize committee to all scientists in the pool. In reality we only have citation rankings for all scientists. This means, however, that we have the ordinal rankings of the prize winners with regard to citation metrics. I use maximum likelihood method to infer the most probable correlation coefficient to produce the observed pattern of ordinal ranks of the prize winners. I get the correlation coefficients of 0.47 and 0.59 between the composite citation indicator and getting Abel Prize a…
More than 30 years after Cronin's seminal paper on "the need for a theory of citing" (Cronin, 1981), the metrics community is once again in need of a new theory, this time one for so-called "altmetrics". Altmetrics, short for alternative (to citation) metrics -- and as such a misnomer -- refers to a new group of metrics based (largely) on social media events relating to scholarly communication. As current definitions of altmetrics are shaped and limited by active platforms, technical possibilities, and business models of aggregators such as Altmetric.com, ImpactStory, PLOS, and Plum Analytics,…
F1000 recommendations have been validated as a potential data source for research evaluation, but reasons for differences between F1000 Article Factor (FFa scores) and citations remain to be explored. By linking 28254 publications in F1000 to citations in Scopus, we investigated the effect of research level and article type on the internal consistency of assessments based on citations and FFa scores. It turns out that research level has little impact, while article type has big effect on the differences. These two measures are significantly different for two groups: non-primary research or evi…
Scholarly and social impacts of scientific publications could be measured by various metrics. In this study, the relationship between various metrics of 63,805 PLOS research articles are studied. Generally, article views correlate well with citation, however, different types of article view have different levels of correlation with citation, when pdf download correlates the citation most significantly. It's necessary for publishers and journals to provide detailed and comprehensive article metrics. Although the low correlation between social attention and citation is confirmed by this study an…
In the past, several works have investigated ways for combining quantitative and qualitative methods in research assessment exercises. In this work, we aim at introducing a methodology to explore whether citation-based metrics, calculated only considering open bibliographic and citation data, can yield insights on how human peer-review of research assessment exercises is conducted. To understand if and what metrics provide relevant information, we propose to use a series of machine learning models to replicate the decisions of the committees of the research assessment exercises.
In this work we propose a metric to assess academic productivity based on publication outputs. We are interested in knowing how well a research group in an area of knowledge is doing relatively to a pre-selected set of reference groups, where each group is composed by academics or researchers. To assess academic productivity we propose a new metric, which we call P-score. Our metric P-score assigns weights to venues using only the publication patterns of selected reference groups. This implies that P-score does not depend on citation data and thus, that it is simpler to compute particularly in…
Citation measures, and newer altmetric measures such as downloads are now commonly used to inform personnel decisions. How well do or can these measures measure or predict the past, current of future scholarly performance of an individual? Using data from the Smithsonian/NASA Astrophysics Data System we analyze the publication, citation, download, and distinction histories of a cohort of 922 individuals who received a U.S. PhD in astronomy in the period 1972-1976. By examining the same and different measures at the same and different times for the same individuals we are able to show the capab…
