A fundamental problem in citation analysis is the prediction of the long-term citation impact of recent publications. We propose a model to predict a probability distribution for the future number of citations of a publication. Two predictors are used: The impact factor of the journal in which a publication has appeared and the number of citations a publication has received one year after its appearance. The proposed model is based on quantile regression. We employ the model to predict the future number of citations of a large set of publications in the field of physics. Our analysis shows that both predictors (i.e., impact factor and early citations) contribute to the accurate prediction of long-term citation impact. We also analytically study the behavior of the quantile regression coefficients for high quantiles of the distribution of citations. This is done by linking the quantile regression approach to a quantile estimation technique from extreme value theory. Our work provides insight into the influence of the impact factor and early citations on the long-term citation impact of a publication, and it takes a step toward a methodology that can be used to assess research institutions based on their most recently published work.
The citation impact of a scientific publication is usually seen as a one-dimensional concept. We introduce a multi-dimensional framework for characterizing the citation impact of a publication. In addition to the level of citation impact, quantified by the number of citations received by a publication, we also conceptualize and operationalize the depth and breadth and the dependence and independence of the citation impact of a publication. The proposed framework distinguishes between publications that have a deep citation impact, typically in a relatively narrow research area, and publications…
We analyzed publications data in WoS and Scopus to compare publications in native languages vs publications in English and find any distinctive patterns. We analyzed their distribution by research areas, languages, type of access and citation patterns. The following trends were found: share of English publications increases over time; native-language publications are read and cited less than English-language outside the origin country; open access impact on views and citation is higher for native languages; journal ranking correlates with the share of English publications for multi-language jo…
The ability to predict the long-term impact of a scientific article soon after its publication is of great value towards accurate assessment of research performance. In this work we test the hypothesis that good predictions of long-term citation counts can be obtained through a combination of a publication's early citations and the impact factor of the hosting journal. The test is performed on a corpus of 123,128 WoS publications authored by Italian scientists, using linear regression models. The average accuracy of the prediction is good for citation time windows above two years, decreases fo…
Citations are used for research evaluation, and it is therefore important to know which factors influence or associate with citation impact of articles. Several citation factors have been studied in the literature. In this study we propose a new factor, topic growth, that no previous study has studied empirically. The growth rate of topics may influence future citation counts because a high growth in a topic means there are more publications citing previous publications in that topic. We construct topics using community detection in a citation network and use a two-part regression model to stu…
This paper defines and explores the direct citations between citing publications (DCCPs) of a publication. We construct an ego-centered citation network for each paper that contains all of its citing papers and itself, as well as the citation relationships among them. By utilizing a large-scale scholarly dataset from the computer science field in the Microsoft Academic Graph (MAG-CS) dataset, we find that DCCPs exist universally in medium and highly cited papers. For those papers that have DCCPs, DCCPs do occur frequently; highly cited papers tend to contain more DCCPs than others. Meanwhile, …
This work explores the distribution of citations for the publications of top scientists. A first objective is to find out whether the 80-20 Pareto rule applies, that is if 80% of the citations to a top scientist's work concern 20% of their publications. Observing that the rule does not apply, we also measure the dispersion of the citation distribution by means of the Gini coefficient. Further, we investigate the question of what share of a top scientist' publications go uncited. Finally, we study the relation between the dispersion of the citation distribution and the share of uncited publicat…
CitNetExplorer被用来研究1998-2012年15年间摩擦学科学出版物之间的引文网络。分析了Web of Science的三个数据集：（1）摩擦学研究的核心出版物，（2）纳米摩擦学出版物和（3）Bharat Bhushan（纳米摩擦学研究的主要贡献者）的出版物。在此基础上，提出了改进CitNetExplorer的建议。
A concern has been raised that "covidization" of research would cause an overemphasizes on covid-19 and pandemics at the expense of other research. The rapid growth of publications related to the Covid-19 pandemic renders a vast amount of citations from this literature. This growth may affect bibliometric indicators. In this paper I explored how the growth of covid-19 publications influences bibliometric indicators commonly used in university rankings, research evaluation and research allocation, namely the field normalized citation score and the journal impact factor. I found that the burst o…
F1000 recommendations as a new data source for research evaluation: A comparison with citations
F1000是生物和医学研究的出版后同行评议服务。F1000旨在推荐生物医学文献中的重要出版物，从这个角度来看，F1000可能是一个有趣的研究评估工具。通过将完整的F1000建议数据库与Web of Science书目数据库相连接，我们能够对F1000建议和引文进行全面的比较。我们发现，生物医学文献中约2%的出版物至少获得一项F1000推荐。推荐出版物平均收到1.30条推荐，90%以上的推荐是在出版物出版后半年内给出的。事实证明，F1000推荐和引用之间存在明显的相关性。然而，这种相关性相对较弱，至少弱于期刊影响力和引用之间的相关性。需要进行更多的研究，以确定在评估出版物的影响方面，建议和引用之间存在差异的主要原因。
In this work, we try to answer the question of which method, peer review vs bibliometrics, better predicts the future overall scholarly impact of scientific publications. We measure the agreement between peer review evaluations of Web of Science indexed publications submitted to the first Italian research assessment exercise and long-term citations of the same publications. We do the same for an early citation-based indicator. We find that the latter shows stronger predictive power, i.e., it more reliably predicts late citations in all the disciplinary areas examined, and for any citation time…
ResearchGate has emerged as a popular professional network for scientists and researchers in a very short span of time. Similar to Google Scholar, the ResearchGate indexing uses an automatic crawling algorithm that extracts bibliographic data, citations and other information about scholarly articles from various sources. However, it has been observed that the two platforms often show different publication and citation data for the same institutions, journals and authors. This paper, therefore, attempts to analyse and measure the differences in publication counts, citations and different metric…
Dimensions was introduced as an alternative bibliometric database to the well-established Web of Science (WoS) and Scopus, however all three databases have fundamental differences in coverage and content, resultant from their owners' indexation philosophies. In light of these differences, we explore here, using a citation network analysis and assessment of normalised citation impact of "duplicate" publications, whether the three databases offer structurally different perspectives of the bibliometric landscape or if they are essentially homogenous substitutes. Our citation network analysis of c…
Co-citation measurements can reveal the extent to which a concept representing a novel combination of existing ideas evolves towards a specialty. The strength of co-citation is represented by its frequency, which accumulates over time. Of interest is whether underlying features associated with the strength of co-citation can be identified. We use the proximal citation network for a given pair of articles (x, y) to compute theta, an a priori estimate of the probability of co-citation between x and y, prior to their first co-citation.Thus, low values for theta reflect pairs of articles for which…
The substantive and practical significance of citation impact differences between institutions: Guidelines for the analysis of percentiles using effect sizes and confidence intervals
Research instruments play significant roles in the construction of scientific knowledge, even though we have only acquired very limited knowledge about their lifecycles from quantitative studies. This paper aims to address this gap by quantitatively examining the citation contexts of an exemplary research instrument, the Diagnostic and Statistical Manual of Mental Disorders (DSM), in full-text psychological publications. We investigated the relationship between the citation contexts of the DSM and its status as a valid instrument being used and described by psychological researchers. We specif…
In this paper we present a first large-scale analysis of the relationship between Mendeley readership and citation counts with particular documents bibliographic characteristics. A data set of 1.3 million publications from different fields published in journals covered by the Web of Science (WoS) has been analyzed. This work reveals that document types that are often excluded from citation analysis due to their lower citation values, like editorial materials, letters, or news items, are strongly covered and saved in Mendeley, suggesting that Mendeley readership can reliably inform the analysis…
This study examines the correlational relationships between local journal authorship, local and external citation counts, full-text downloads, link-resolver clicks, and four global journal impact factor indices within an all-disciplines journal collection of 12,200 titles and six subject subsets at the University of Illinois at Urbana-Champaign (UIUC) Library. While earlier investigations of the relationships between usage (downloads) and citation metrics have been inconclusive, this study shows strong correlations in the all-disciplines set and most subject subsets. The normalized Eigenfactor…
Regional and Global Science: Latin American and Caribbean publications in the SciELO Citation Index and the Web of Science
我们比较了拉丁美洲和加勒比地区（LAC）出版物在Web of Science（WoS）的核心馆藏索引中的可见性-科学引文索引扩展，社会科学引文索引和艺术与人文学科引文索引-以及SciELO引文索引（SciELO CI），该索引于2014年集成到更大的WoS平台中。这种比较的目的是帮助我们了解拉丁美洲和加勒比地区产生的科学知识的传播，并提供一些思考的潜在好处的衔接区域索引工作到WoS更好地了解地理和学科的贡献。SciELO CI的区域水平与WoS的全球范围有何关系？在WoS中，LAC作者在全球范围内被纳入国际网络，而SciELO为LAC研究人员提供了一个互动平台。SciELO与WoS的结合可能会提高区域期刊的国际知名度，但代价是独立的期刊纳入标准。
This work analyzes the variation over time of the effect of geographic distance on knowledge flows. The flows are measured through the citations exchanged between scientific publications, including and excluding self-citations. To calculate geographic distances between citing and cited publication, each publication is associated with a "prevailing" territory, according to the authors' affiliations. We then apply a gravity model to account for the research size of the territories, in terms of cognitive proximity of citing-cited publications. The field of observation is the 2010-2017 world publi…
Studying the history of research fields by analyzing publication records and topical and/or keyword searches with a full Reference Publication Year Spectroscopy (RPYS) has been introduced as a powerful tool to identify the corresponding root publications. However, for a rather new and interdisciplinary research field like Solar Energy Meteorology (SEM), this method is not feasible to get a reasonably exhaustive publication set. Therefore we apply its variant RPYS-CO to all publications co-cited with one highly important marker paper, using the CRExplorer for plotting and inspecting the spectro…
The growing complexity of scientific challenges demands increasingly intense research collaboration, both domestic and international. The resulting trend affects not only the modes of producing new knowledge, but also the way it is disseminated within scientific communities. This paper analyses the relationship between the "degree of internationalization" of a country's scientific production and that of the relevant citing publications. The empirical analysis is based on 2010-2012 Italian publications. Findings show: i) the probability of being cited increases with the degree of internationali…
Citation impact is commonly assessed using direct, first-order citation relations. We consider here instead the indirect influence of publications on new publications via citations. We present a novel method to quantify the higher-order citation influence of publications, considering both direct, or first-order, and indirect, or higher-order citations. In particular, we are interested in higher-order citation influence at the level of disciplines. We apply this method to the whole Web of Science data at the level of disciplines. We find that a significant amount of influence -- 42% -- stems fr…
In course of the organization of Workshop III entitled "Cited References Analysis Using CRExplorer" at the International Conference of the International Society for Scientometrics and Informetrics (ISSI2021), we have prepared three reference publication year spectroscopy (RPYS) analyses: (i) papers published in Journal of Informetrics; (ii) papers regarding the topic altmetrics; and (iii) papers published by Ludo Waltman (we selected this researcher since he received the Derek de Solla Price Memorial Medal during the ISSI2021 conference). The first RPYS analysis has been presented live at the …
The effects of Open Access (OA) upon journal performance are investigated. The key research question holds: How does the citation impact and publication output of journals switching ('flipping') from non-OA to Gold-OA develop after their switch to Gold-OA? A review is given of the literature, with an emphasis on studies dealing with flipping journals. Two study sets with 119 and 100 flipping journals, derived from two different OA data sources (DOAJ and OAD), are compared with two control groups, one based on a standard bibliometric criterion, and a second controlling for a journal's national …
The prediction of the long-term impact of a scientific article is challenging task, addressed by the bibliometrician through resorting to a proxy whose reliability increases with the breadth of the citation window. In the national research assessment exercises using metrics the citation window is necessarily short, but in some cases is sufficient to advise the use of simple citations. For the Italian VQR 2011-2014, the choice was instead made to adopt a linear weighted combination of citations and journal metric percentiles, with weights differentiated by discipline and year. Given the strateg…
Empirical evidence shows that co-authored publications achieve higher visibility and impact. The aim of the current work is to test for the existence of a similar correlation for Italian publications. We also verify if such correlation differs: i) by subject category and macro-area; ii) by document type; iii) over the course of time. The results confirm world-level evidence, showing a consistent and significant linear growth in the citability of a publication with number of co-authors, in almost all subject categories. The effects are more remarkable in the fields of Social Sciences and Art & …
With the passage of more time from the original date of publication, the measure of the impact of scientific works using subsequent citation counts becomes more accurate. However the measurement of individual and organizational research productivity should ideally refer to a period with closing date just prior to the evaluation exercise. Therefore it is necessary to compromise between accuracy and timeliness. This work attempts to provide an order of magnitude for the error in measurement that occurs with decreasing the time lapse between date of publication and citation count. The analysis is…
In research policy, effective measures that lead to improvements in the generation of knowledge must be based on reliable methods of research assessment, but for many countries and institutions this is not the case. Publication and citation analyses can be used to estimate the part played by countries and institutions in the global progress of knowledge, but a concrete method of estimation is far from evident. The challenge arises because publications that report real progress of knowledge form an extremely low proportion of all publications; in most countries and institutions such contributio…
This work applies a new approach to measure knowledge flows. Assuming that citation linkages between articles imply a flow of knowledge from the cited to the citing authors, we investigate the geographic flows of scientific knowledge produced in Italy across its regions, at both overall and field level. Furthermore, we measure the the specialization indexes for outflows and inflows of knowledge by a given region. Findings show that larger regions in terms of research output are more likely net exporters of new knowledge. At the same time, we register a positive correlation between the share of…
In this work we ask whether and to what extent applying a predictor of publications' impact better than early citations, has an effect on the assessment of research performance of individual scientists. Specifically, we measure the total impact of Italian professors in the sciences and economics in a period of time, valuing their publications first by early citations and then by a weighted combination of early citations and impact factor of the hosting journal. As expected, scores and ranks by the two indicators show a very strong correlation, but there occur also significant shifts in many fi…
Mendeley readership as a filtering tool to identify highly cited publications
这项研究提出了一个大规模的分析，随着时间的推移和跨学科的Mendeley读者分数的分布和存在。我们研究是否Mendeley读者评分（RS）可以识别高引用的出版物更有效地比期刊引用分数（JCS）。分析了2004-2013年期间在5个主要科学领域发表的具有DOI的Web of Science（WoS）出版物。这项研究的主要结果表明，读者评分比期刊引用评分更有效（在精确度/召回值方面），以识别所有科学领域和出版年份的高引用出版物。调查结果还显示，所有出版物中有86.5%被Mendeley覆盖，并且至少有一位读者。此外，具有Mendeley读者分数的出版物的份额从2004年的84%增加到2009年的89%，从2010年的88%下降到2013年的82%。然而，值得注意的是，从2010年起，出版物的读者密度平均高于引用分数。这表明，与引用分数相比，读者分数对于最近的出版物更普遍，因此它们可以作为研究影响的早期指标。这些发现突出了Mendeley作为科学计量学工具的潜力和价值，特别是作为识别高引用出版物的相关工具。
Academic publications have been evaluated in terms of their impact on research communities based on many metrics, such as the number of citations. On the other hand, the impact of academic publications on industry has been rarely studied. This paper investigates how academic publications contribute to software development by analyzing publication citations in source code comments in open source software repositories. We propose an automated approach for detecting academic publications based on Named Entity Recognition, and achieve 0.90 in $F_1$ as detection accuracy. We conduct a large-scale s…
In this study, I analyze the DBLP bibliographic database to study role of single author publications in the computer science literature between 1940 and 2019. I examine the demographics and reception by computing the population fraction, citation statistics, and PageRank scores of single author publications over the years. Both the population fraction and reception have been continuously declining since the 1940s. The overall decaying trend of single author publications is qualitatively consistent with those observed in other scientific disciplines, though the diminution is taking place severa…
We present a large-scale analysis of the accuracy of citation data in the Web of Science and Scopus databases. The analysis is based on citations given in publications in Elsevier journals. We reveal significant data quality problems for both databases. Missing and incorrect references are important problems in Web of Science. Duplicate publications are a serious problem in Scopus.
World University rankings have become well-established tools that students, university managers and policy makers read and use. Each ranking claims to have a unique methodology capable of measuring the 'quality' of universities. The purpose of this paper is to analyze to which extent these different rankings measure the same phenomenon and what it is that they are measuring. For this, we selected a total of seven world-university rankings and performed a principal component analysis. After ensuring that despite their methodological differences, they all come together to a single component, we …
Digital Science's Dimensions is envisaged as a next-generation research and discovery platform for a better and more efficient access to cross-referenced scholarly publications, grants, patents, and clinical trials. As a new addition to the growing open citation resources, it offers opportunities that may benefit a wide variety of stakeholders of scientific publications from researchers, policy makers, and the general public. In this article, we explore and demonstrate some of the practical potentials in terms of cascading citation expansions. Given a set of publications, the cascading citatio…
