A principal component analysis of 39 scientific impact measures
Johan Bollen, Herbert Van de Sompel, Aric Hagberg, Ryan Chute
发表信息：
Bollen J, Van de Sompel H, Hagberg A, Chute R, 2009 A Principal Component Analysis of 39 Scientific Impact Measures. PLoS ONE 4(6): e6022. (http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0006022)
The impact of scientific publications has traditionally been expressed in terms of citation counts. However, scientific activity has moved online over the past decade. To better capture scientific impact in the digital era, a variety of new impact measures has been proposed on the basis of social network analysis and usage log data. Here we investigate how these new measures relate to each other, and how accurately and completely they express scientific impact. We performed a principal component analysis of the rankings produced by 39 existing and proposed measures of scholarly impact that were calculated on the basis of both citation and usage log data. Our results indicate that the notion of scientific impact is a multi-dimensional construct that can not be adequately measured by any single indicator, although some measures are more suitable than others. The commonly used citation Impact Factor is not positioned at the core of this construct, but at its periphery, and should thus be used with caution.
We consider distributions of scientific journals impact factor. Analysing 9028 scientific journals with the largest impact factors, we found that the distribution of them is year-to-year stable (at least for analysed 2011-2013 years), and it has the character of the exponential Boltzmann distribution with the power law asymptotic (tail).
Nowadays impact factor is the significant indicator for journal evaluation. In impact factor calculation is used number of all citations to journal, regardless of the prestige of cited journals, however, scientific units (paper, researcher, journal or scientific organization) cited by journals with high impact factor or researchers with high Hirsch index are more important than objects cited by journals without impact factor or unknown researcher. In this paper was offered weighted impact factor for getting more accurate rankings for journals, which consider not only quantity of citations, but…
The paper comments on "Quantifying long-term scientific impact". It indicates that there is a mistake of [D. S. Wang , C. Song, A. L. Barabasi, Quantifying long-term scientific impact, Science 342, 127 (2013), arXiv:1306.3293].
In this paper we consider the scientific and career performance of principal investigators (PI's) of publicly funded research projects compared to scientific performance of all researchers. Our study is based on high quality data about (i) research projects awarded in Slovenia in the period 1994-2016 (7508 projects with 2725 PI's in total) and (ii) about scientific productivity of all researchers in Slovenia that were active in the period 1970-2016 - there are 19598 such researchers in total, including the PI's. We compare average productivity, collaboration, internationality and interdiscipli…
An ongoing interest in innovation studies is to understand how knowledge generated from scientific research can be used in the development of technologies. While previous inquiries have devoted to studying the scientific capacity of technologies and institutional factors facilitating technology transfer, little is known about the intrinsic characteristics of scientific publications that gain direct technological impact. Here we focus on two features, namely basicness and novelty. Using a corpus of 3.8 million papers published between 1980 and 1999, we find that basic science papers and novel p…
Background: Conferences bring scientists together and provide one of the most timely means for disseminating new ideas and cutting-edge works.The importance of conferences in scientific areas is testified by quantitative indicators. In Computer Science, for instance, almost two out of three papers published on Scopus are conference papers. Objective/Purpose: The main goal of this paper is to investigate a novel research question: is there any correlation between the impact of a scientific conference and the venue where it took place? Approach: In order to measure the impact of conferences we c…
The citation impact of a scientific publication is usually seen as a one-dimensional concept. We introduce a multi-dimensional framework for characterizing the citation impact of a publication. In addition to the level of citation impact, quantified by the number of citations received by a publication, we also conceptualize and operationalize the depth and breadth and the dependence and independence of the citation impact of a publication. The proposed framework distinguishes between publications that have a deep citation impact, typically in a relatively narrow research area, and publications…
The aggregated citation relations among journals included in the Science Citation Index provide us with a huge matrix which can be analyzed in various ways. Using principal component analysis or factor analysis, the factor scores can be used as indicators of the position of the cited journals in the citing dimensions of the database. Unrotated factor scores are exact, and the extraction of principal components can be made stepwise since the principal components are independent. Rotation may be needed for the designation, but in the rotated solution a model is assumed. This assumption can be le…
The growth rate of the number of scientific publications is constantly increasing, creating important challenges in the identification of valuable research and in various scholarly data management applications, in general. In this context, measures which can effectively quantify the scientific impact could be invaluable. In this work, we present BIP! DB, an open dataset that contains a variety of impact measures calculated for a large collection of more than 100 million scientific publications from various disciplines.
Scholars, policymakers, and research sponsors have long sought to understand the conditions under which scientific research is used in the policymaking process. Recent research has identified a resource that can be used to trace the use of science across time and many policy domains. US federal agencies are mandated by executive order to justify all economically significant regulations by regulatory impact analyses (RIAs), in which they present evidence of the scientific underpinnings and consequences of the proposed rule. To gain new insight into when and how regulators invoke science in thei…
The issue of duplicate publications has received a lot of attention in the medical literature, but much less in the information science community. This paper aims at analyzing the prevalence and scientific impact of duplicate publications across all fields of research between 1980 and 2007, using a definition of duplicate papers based on their metadata. It shows that in all fields combined, the prevalence of duplicates is one out of two-thousand papers, but is higher in the natural and medical sciences than in the social sciences and humanities. A very high proportion (>85%) of these papers ar…
A widely used measure of scientific impact is citations. However, due to their heavy-tailed distribution, citations are fundamentally difficult to predict. Instead, to characterize scientific impact, we address two analogous questions asked by many scientific researchers: "How will my h-index evolve over time, and which of my previously or newly published papers will contribute to it?" To answer these questions, we perform two related tasks. First, we develop a model to predict authors' future h-indices based on their current scientific impact. Second, we examine the factors that drive papers-…
Nanjing University of Science and Technology（南京理工大学）；Indiana University（印第安纳大学）；Shandong Normal University（山东师范大学）；Nanjing University（南京大学）；Wuhan University（武汉大学）；Université de Montréal（蒙特利尔大学）
The number of publications and the number of citations received have become the most common indicators of scholarly success. In this context, scientific writing increasingly plays an important role in scholars' scientific careers. To understand the relationship between scientific writing and scientific impact, this paper selected 12 variables of linguistic complexity as a proxy for depicting scientific writing. We then analyzed these features from 36,400 full-text Biology articles and 1,797 full-text Psychology articles. These features were compared to the scientific impact of articles, groupe…
This paper provides the first historical analysis of the relationship between collaboration and scientific impact, using three indicators of collaboration (number of authors, number of addresses, and number of countries) and including articles published between 1900 and 2011. The results demonstrate that an increase in the number of authors leads to an increase in impact--from the beginning of the last century onwards--and that this is not simply due to self-citations. A similar trend is also observed for the number of addresses and number of countries represented in the byline of an article. …
Traditionally, the number of citations that a scholarly paper receives from other papers is used as the proxy of its scientific impact. Yet citations can come from domains outside the scientific community, and one such example is through patented technologies---paper can be cited by patents, achieving technological impact. While the scientific impact of papers has been extensively studied, the technological aspect remains less known in the literature. Here we aim to fill this gap by presenting a comparative study on how 919 thousand biomedical papers are cited by U.S. patents and by other pape…
In this paper, we propose a measure to assess scientific impact that discounts self-citations and does not require any prior knowledge on the their distribution among publications. This index can be applied to both researchers and journals. In particular, we show that it fills the gap of h-index and similar measures that do not take into account the effect of self-citations for authors or journals impact evaluation. The paper provides with two real-world examples: in the former, we evaluate the research impact of the most productive scholars in Computer Science (according to DBLP); in the latt…
The impact factor (IF) of scientific journals has acquired a major role in the evaluations of the output of scholars, departments and whole institutions. Typically papers appearing in journals with large values of the IF receive a high weight in such evaluations. However, at the end of the day one is interested in assessing the impact of individuals, rather than papers. Here we introduce Author Impact Factor (AIF), which is the extension of the IF to authors. The AIF of an author A in year $t$ is the average number of citations given by papers published in year $t$ to papers published by A in …
The impact factor has been extensively used in the last years to assess journals visibility and prestige. While the impact factor is useful to compare journals, the specificities of subfields visibility in journals are overlooked whenever visibility is measured only at the journal level. In this paper, we analyze the subfields visibility in a subset of over 450,000 Physics papers. We show that the visibility of subfields is not regular in the considered dataset. In particular years, the variability in subfields impact factor in a journal reached 75% of the average subfields impact factor. We a…
The lack of predictability of citation-based measures frequently used to gauge impact, from impact factors to short-term citations, raises a fundamental question: Is there long-term predictability in citation patterns? Here, we derive a mechanistic model for the citation dynamics of individual papers, allowing us to collapse the citation histories of papers from different journals and disciplines into a single curve, indicating that all papers tend to follow the same universal temporal pattern. The observed patterns not only help us uncover basic mechanisms that govern scientific impact but al…
Over the last few years, several major scientific fraud cases have shocked the scientific community. The number of retractions each year has also increased tremendously, especially in the biomedical field, and scientific misconduct accounts for approximately more than half of those retractions. It is assumed that co-authors of retracted papers are affected by their colleagues' misconduct, and the aim of this study is to provide empirical evidence of the effect of retractions in biomedical research on co-authors' research careers. Using data from the Web of Science (WOS), we measured the produc…
Comments are special types of publications whose aim is to correct or criticize previously published papers. For this reason, comments are believed to make commented papers less worthy or trusty to the eyes of the scientific community, and thus predestined to have low scientific impact. Here, we show that such belief is not supported by empirical evidence. We consider thirteen major publication outlets in science, and perform systematic comparisons between the citations accumulated by commented and non commented articles. We find that (i) commented papers are, on average, much more cited than …
This paper analyzes the effect of interdisciplinarity on the scientific impact of individual papers. Using all the papers published in Web of Science in 2000, we define the degree of interdisciplinarity of a given paper as the percentage of its cited references made to journals of other disciplines. We show that, although for all disciplines combined there is no clear correlation between the level of interdisciplinarity of papers and their citation rates, there are nonetheless some disciplines in which a higher level of interdisciplinarity is related to a higher citation rates. For other disci…
We performed a citation analysis on the Web of Science publications consisting of more than 63 million articles and 1.45 billion citations on 254 subjects from 1981 to 2020. We proposed the Article's Scientific Prestige (ASP) metric and compared this metric to number of citations (#Cit) and journal grade in measuring the scientific impact of individual articles in the large-scale hierarchical and multi-disciplined citation network. In contrast to #Cit, ASP, that is computed based on the eigenvector centrality, considers both direct and indirect citations, and provides steady-state evaluation c…
Problems for evaluation and impact of published scientific works and their authors are discussed. The role of citations in this process is pointed out. Different bibliometric indicators are reviewed in this connection and ways for generation of new bibliometric indices are given. The influence of different circumstances, like self-citations, number of authors, time dependence and publication types, on the evaluation and impact of scientific papers are considered. The repercussion of works citations and their content is investigated in this respect. Attention is paid also on implicit citations …
Inspired by the social and economic benefits of diversity, we analyze over 9 million papers and 6 million scientists to study the relationship between research impact and five classes of diversity: ethnicity, discipline, gender, affiliation, and academic age. Using randomized baseline models, we establish the presence of homophily in ethnicity, gender and affiliation. We then study the effect of diversity on scientific impact, as reflected in citations. Remarkably, of the classes considered, ethnic diversity had the strongest correlation with scientific impact. To further isolate the effects o…
h-index has become the most popular indicator for quantifying a scientist's scientific impact in various scientific fields. h-index is defined as the largest number of papers with citation number larger than or equal to h and it treats each citation equally. However, different citations usually come from different papers with different influence and quality, and a citation from a highly influential paper is a greater recognition of the target paper than a citation from an ordinary paper. Based on this assumption, we proposed a new indicator named x-index to quantify a scientist's scientific im…
The empirical and theoretical justification of Gartner hype curves is a very relevant open question in the field of Technological Life Cycle analysis. The scope of the present paper is to introduce a simple model describing the growth of scientific/technological research impact, in the specific case where science is the main source of a new idea driving a technological development, leading to hype-type evolution curves. The main idea of the model is that, in a first stage, the growth of the scientific interest of a new specific field (as can be measured by publication numbers) basically follow…
The constantly increasing rate at which scientific papers are published makes it difficult for researchers to identify papers that currently impact the research field of their interest. Hence, approaches to effectively identify papers of high impact have attracted great attention in the past. In this work, we present a method that seeks to rank papers based on their estimated short-term impact, as measured by the number of citations received in the near future. Similar to previous work, our method models a researcher as she explores the paper citation network. The key aspect is that we incorpo…
The Effect of Gender Diversity on Scientific Team Impact: A Team Roles Perspective
Anhui University（安徽大学）；Yonsei University（延世大学）；Nanjing University（南京大学）；Central China Normal University（华中师范大学）；Hohai University（河海大学）；Nanjing University of Science and Technology（南京理工大学）
The broad coverage of the search for the Higgs boson in the mainstream media is a relative novelty for high energy physics (HEP) research, whose achievements have traditionally been limited to scholarly literature. This paper illustrates the results of a scientometric analysis of HEP computing in scientific literature, institutional media and the press, and a comparative overview of similar metrics concerning representative particle physics measurements. The picture emerging from these scientometric data documents the scientific impact and social perception of HEP computing. The results of thi…
A well established agenda on the research output, impact, and structure of global scientific elites such as Nobel Prize laureates has generated interest in the scientific elites from developing countries. This study deploys science mapping techniques to provide a comprehensive analysis of the output, impact, and structure of the Colombian scientific elite, i.e., researchers awarded with the Alejandro Angel Escobar Foundation National Prize 1990 2020, known locally as the Colombian Nobel. Findings showed that the Colombian scientific elite has a broader agenda than indexing titles in internatio…
In the context of altmetrics, tweets have been discussed as potential indicators of immediate and broader societal impact of scientific documents. However, it is not yet clear to what extent Twitter captures actual research impact. A small case study (Thelwall et al., 2013b) suggests that tweets to journal articles neither comment on nor express any sentiments towards the publication, which suggests that tweets merely disseminate bibliographic information, often even automatically. This study analyses the sentiments of tweets for a large representative set of scientific papers by specifically …
Citations are commonly held to represent scientific impact. To date, however, there is no empirical evidence in support of this postulate that is central to research assessment exercises and Science of Science studies. Here, we report on the first empirical verification of the degree to which citation numbers represent scientific impact as it is actually perceived by experts in their respective field. We run a large-scale survey of about 2000 corresponding authors who performed a pairwise impact assessment task across more than 20000 scientific articles. Results of the survey show that citatio…
Government funding agencies and foundations tend to perceive novelty as necessary for scientific impact and hence prefer to fund novel instead of incremental projects. Evidence linking novelty and the eventual impact of a grant is surprisingly scarce, however. Here, we examine this link by analyzing 920,000 publications funded by 170,000 grants from the National Science Foundation (NSF) and the National Institutes of Health (NIH) between 2008 and 2016. We use machine learning to quantify grant novelty at the time of funding and relate that measure to the citation dynamics of these publications…
The author of this letter to the editor would like to set forth the argument that scientometrics is currently in a phase in which a taxonomic change, and hence a revolution, is taking place. One of the key terms in scientometrics is scientific impact which nowadays is understood to mean not only the impact on science but the impact on every area of society.
Attempts to understand the consequence of any individual scientist's activity within the long-term trajectory of science is one of the most difficult questions within the philosophy of science. Because scientific publications play such as central role in the modern enterprise of science, bibliometric techniques which measure the ``impact'' of an individual publication as a function of the number of citations it receives from subsequent authors have provided some of the most useful empirical data on this question. Until recently, Thompson/ISI has provided the only source of large-scale ``invert…
University of Electronic Science and Technology of China（电子科技大学）；Iowa State University（爱荷华州立大学）；University of Maryland（马里兰大学）
Quantifying and predicting the long-term impact of scientific writings or individual scholars has important implications for many policy decisions, such as funding proposal evaluation and identifying emerging research fields. In this work, we propose an approach based on Heterogeneous Dynamical Graph Neural Network (HDGNN) to explicitly model and predict the cumulative impact of papers and authors. HDGNN extends heterogeneous GNNs by incorporating temporally evolving characteristics and capturing both structural properties of attributed graph and the growing sequence of citation behavior. HDGN…
We investigate how textual properties of scientific papers relate to the number of citations they receive. Our main finding is that correlations are non-linear and affect differently most-cited and typical papers. For instance, we find that in most journals short titles correlate positively with citations only for the most cited papers, for typical papers the correlation is in most cases negative. Our analysis of 6 different factors, calculated both at the title and abstract level of 4.3 million papers in over 1500 journals, reveals the number of authors, and the length and complexity of the a…
There exist ample demonstrations that indicators of scholarly impact analogous to the citation-based ISI Impact Factor can be derived from usage data. However, contrary to the ISI IF which is based on citation data generated by the global community of scholarly authors, so far usage can only be practically recorded at a local level leading to community-specific assessments of scholarly impact that are difficult to generalize to the global scholarly community. We define a journal Usage Impact Factor which mimics the definition of the Thomson Scientific's ISI Impact Factor. Usage Impact Factor r…
Beyond Single-Dimension Novelty: How Combinations of Theory, Method, and Results-based Novelty Shape Scientific Impact
School of Management, Anhui University（安徽大学管理学院）；Department of Information Management, Nanjing University of Science and Technology（南京理工大学信息管理学院）；School of Information Management, Central China Normal University（华中师范大学信息管理学院）
Using bibliometric data artificially generated through a model of citation dynamics calibrated on empirical data, we compare several indicators for the scientific impact of individual researchers. The use of such a controlled setup has the advantage of avoiding the biases present in real databases, and allows us to assess which aspects of the model dynamics and which traits of individual researchers a particular indicator actually reflects. We find that the simple citation average performs well in capturing the intrinsic scientific ability of researchers, whatever the length of their career. O…
Throughout history, a relatively small number of individuals have made a profound and lasting impact on science and society. Despite long-standing, multi-disciplinary interests in understanding careers of elite scientists, there have been limited attempts for a quantitative, career-level analysis. Here, we leverage a comprehensive dataset we assembled, allowing us to trace the entire career histories of nearly all Nobel laureates in physics, chemistry, and physiology or medicine over the past century. We find that, although Nobel laureates were energetic producers from the outset, producing wo…
The present study aims to analyze relationship between Citations Normalized Score (NCS) of scientific publications and Article Processing Charges (APCs) amounts of Gold Open access publications. To do so, we use APCs information provided by OpenAPC database and citations scores of publications in the Web of Science database (WoS). Database covers the period from 2006 to 2019 with 83,752 articles published in 4751 journals belonging to 267 distinct publishers. Results show that contrary to this belief, paying dearly does not necessarily increase the impact of publications. First, large publishe…
This study utilizes global digitalized books and articles to examine the scientific fame of the most influential physicists. Our research reveals that the greatest minds are gone but not forgotten. Their scientific impacts on human history have persisted for centuries. We also find evidence in support of own-group fame preference, i.e., that the scientists have greater reputations in their home countries or among scholars sharing the same languages. We argue that, when applied appropriately, Google Books and Ngram Viewer can serve as promising tools for altmetrics, providing a more comprehensi…
