We investigate the similarities of pairs of articles which are co-cited at the different co-citation levels of the journal, article, section, paragraph, sentence and bracket. Our results indicate that textual similarity, intellectual overlap (shared references), author overlap (shared authors), proximity in publication time all rise monotonically as the co-citation level gets lower (from journal to bracket). While the main gain in similarity happens when moving from journal to article co-citation, all level changes entail an increase in similarity, especially section to paragraph and paragraph to sentence/bracket levels. We compare results from four journals over the years 2010-2015: Cell, the European Journal of Operational Research, Physics Letters B and Research Policy, with consistent general outcomes and some interesting differences. Our findings motivate the use of granular co-citation information as defined by meaningful units of text, with implications for, among others, the elaboration of maps of science and the retrieval of scholarly literature.
The citation impact of a scientific publication is usually seen as a one-dimensional concept. We introduce a multi-dimensional framework for characterizing the citation impact of a publication. In addition to the level of citation impact, quantified by the number of citations received by a publication, we also conceptualize and operationalize the depth and breadth and the dependence and independence of the citation impact of a publication. The proposed framework distinguishes between publications that have a deep citation impact, typically in a relatively narrow research area, and publications…
A fundamental problem in citation analysis is the prediction of the long-term citation impact of recent publications. We propose a model to predict a probability distribution for the future number of citations of a publication. Two predictors are used: The impact factor of the journal in which a publication has appeared and the number of citations a publication has received one year after its appearance. The proposed model is based on quantile regression. We employ the model to predict the future number of citations of a large set of publications in the field of physics. Our analysis shows tha…
We analyzed publications data in WoS and Scopus to compare publications in native languages vs publications in English and find any distinctive patterns. We analyzed their distribution by research areas, languages, type of access and citation patterns. The following trends were found: share of English publications increases over time; native-language publications are read and cited less than English-language outside the origin country; open access impact on views and citation is higher for native languages; journal ranking correlates with the share of English publications for multi-language jo…
The ability to predict the long-term impact of a scientific article soon after its publication is of great value towards accurate assessment of research performance. In this work we test the hypothesis that good predictions of long-term citation counts can be obtained through a combination of a publication's early citations and the impact factor of the hosting journal. The test is performed on a corpus of 123,128 WoS publications authored by Italian scientists, using linear regression models. The average accuracy of the prediction is good for citation time windows above two years, decreases fo…
Citations are used for research evaluation, and it is therefore important to know which factors influence or associate with citation impact of articles. Several citation factors have been studied in the literature. In this study we propose a new factor, topic growth, that no previous study has studied empirically. The growth rate of topics may influence future citation counts because a high growth in a topic means there are more publications citing previous publications in that topic. We construct topics using community detection in a citation network and use a two-part regression model to stu…
This paper defines and explores the direct citations between citing publications (DCCPs) of a publication. We construct an ego-centered citation network for each paper that contains all of its citing papers and itself, as well as the citation relationships among them. By utilizing a large-scale scholarly dataset from the computer science field in the Microsoft Academic Graph (MAG-CS) dataset, we find that DCCPs exist universally in medium and highly cited papers. For those papers that have DCCPs, DCCPs do occur frequently; highly cited papers tend to contain more DCCPs than others. Meanwhile, …
This work explores the distribution of citations for the publications of top scientists. A first objective is to find out whether the 80-20 Pareto rule applies, that is if 80% of the citations to a top scientist's work concern 20% of their publications. Observing that the rule does not apply, we also measure the dispersion of the citation distribution by means of the Gini coefficient. Further, we investigate the question of what share of a top scientist' publications go uncited. Finally, we study the relation between the dispersion of the citation distribution and the share of uncited publicat…
In this study, the academic status of users of scientific publications in Mendeley is explored in order to analyse the usage pattern of Mendeley users in terms of subject fields, citation and readership impact. The main focus of this study is on studying the filtering capacity of Mendeley readership counts compared to journal citation scores in detecting highly cited WoS publications. Main finding suggests a faster reception of Mendeley readerships as compared to citations across 5 major field of science. The higher correlations of scientific users with citations indicate the similarity betwee…
In this study, the presence and distribution of both Mendeley readerships and Web of Science citations for the publications published in the 43 Iranian international journals indexed in Journal Citation Reports have been investigated. The aim was to determine the impact, visibility and use of the publications published by the Iranian international journals in Mendeley compared to their citation impact; furthermore, to explore if there is any relation between these two impact indicators (Mendeley readership counts and WoS citation counts) for these publications. The DOIs of the 1,884 publicatio…
The growing focus in research policy worldwide on top scientists makes it increasingly important to define adequate supporting measures to help identify excellent scientists. Highly cited publications have since long been associated to research excellence. At the same time, the analysis of the high-end of citation distributions still is a challenging topic in evaluative bibliometrics. Evaluations typically require indicators that generate sufficiently stable results when applied to recent publication records of limited size. Highly cited publications have been identified using two techniques i…
We analysed 580 articles (original articles only) published in Medical Journal of Malaysia between 2004 and 2008, the resources referenced by the articles and the citations and impact received. Our aim was to examine article and author productivity, the age of references used and impact of the journal. Publication data was obtained from MyAIS database and Google Scholar provided the citation data. From the 580 articles analyzed, contributors mainly come from the hospitals, universities and clinics. Contributions from foreign authors are low. The useful lives of references cited were between 3 …
CitNetExplorer has been used to study the citation networks among the scientific publications on tribology during the 15 years period from 1998-2012. Three data sets from Web of Science have been analyzed: (1) Core publications of tribology research, (2) publications on nanotribology and (3) publications of Bharat Bhushan (a top-contributor to nanotribology research). Based on this study, some suggestions are made to improve the CitNetExplorer.
A concern has been raised that "covidization" of research would cause an overemphasizes on covid-19 and pandemics at the expense of other research. The rapid growth of publications related to the Covid-19 pandemic renders a vast amount of citations from this literature. This growth may affect bibliometric indicators. In this paper I explored how the growth of covid-19 publications influences bibliometric indicators commonly used in university rankings, research evaluation and research allocation, namely the field normalized citation score and the journal impact factor. I found that the burst o…
An extensive analysis of the presence of different altmetric indicators provided by Altmetric.com across scientific fields is presented, particularly focusing on their relationship with citations. Our results confirm that the presence and density of social media altmetric counts are still very low and not very frequent among scientific publications, with 15%-24% of the publications presenting some altmetric activity and concentrating in the most recent publications, although their presence is increasing over time. Publications from the social sciences, humanities and the medical and life scien…
F1000 is a post-publication peer review service for biological and medical research. F1000 aims to recommend important publications in the biomedical literature, and from this perspective F1000 could be an interesting tool for research evaluation. By linking the complete database of F1000 recommendations to the Web of Science bibliographic database, we are able to make a comprehensive comparison between F1000 recommendations and citations. We find that about 2% of the publications in the biomedical literature receive at least one F1000 recommendation. Recommended publications on average receiv…
In this work, we try to answer the question of which method, peer review vs bibliometrics, better predicts the future overall scholarly impact of scientific publications. We measure the agreement between peer review evaluations of Web of Science indexed publications submitted to the first Italian research assessment exercise and long-term citations of the same publications. We do the same for an early citation-based indicator. We find that the latter shows stronger predictive power, i.e., it more reliably predicts late citations in all the disciplinary areas examined, and for any citation time…
ResearchGate has emerged as a popular professional network for scientists and researchers in a very short span of time. Similar to Google Scholar, the ResearchGate indexing uses an automatic crawling algorithm that extracts bibliographic data, citations and other information about scholarly articles from various sources. However, it has been observed that the two platforms often show different publication and citation data for the same institutions, journals and authors. This paper, therefore, attempts to analyse and measure the differences in publication counts, citations and different metric…
Dimensions was introduced as an alternative bibliometric database to the well-established Web of Science (WoS) and Scopus, however all three databases have fundamental differences in coverage and content, resultant from their owners' indexation philosophies. In light of these differences, we explore here, using a citation network analysis and assessment of normalised citation impact of "duplicate" publications, whether the three databases offer structurally different perspectives of the bibliometric landscape or if they are essentially homogenous substitutes. Our citation network analysis of c…
Clustering scientific publications in an important problem in bibliometric research. We demonstrate how two software tools, CitNetExplorer and VOSviewer, can be used to cluster publications and to analyze the resulting clustering solutions. CitNetExplorer is used to cluster a large set of publications in the field of astronomy and astrophysics. The publications are clustered based on direct citation relations. CitNetExplorer and VOSviewer are used together to analyze the resulting clustering solutions. Both tools use visualizations to support the analysis of the clustering solutions, with CitN…
Co-citation measurements can reveal the extent to which a concept representing a novel combination of existing ideas evolves towards a specialty. The strength of co-citation is represented by its frequency, which accumulates over time. Of interest is whether underlying features associated with the strength of co-citation can be identified. We use the proximal citation network for a given pair of articles (x, y) to compute theta, an a priori estimate of the probability of co-citation between x and y, prior to their first co-citation.Thus, low values for theta reflect pairs of articles for which…
Most publication and citation indicators are based on datasets with multi-authored publications and thus a change in counting method will often change the value of an indicator. Therefore it is important to know why a specific counting method has been applied. I have identified arguments for counting methods in a sample of 32 bibliometric studies published in 2016 and compared the result with discussions of arguments for counting methods in three older studies. Based on the underlying logics of the arguments I have arranged the arguments in four groups. Group 1 focuses on arguments related to …
In our chapter we address the statistical analysis of percentiles: How should the citation impact of institutions be compared? In educational and psychological testing, percentiles are already used widely as a standard to evaluate an individual's test scores - intelligence tests for example - by comparing them with the percentiles of a calibrated sample. Percentiles, or percentile rank classes, are also a very suitable method for bibliometrics to normalize citations of publications in terms of the subject category and the publication year and, unlike the mean-based indicators (the relative cit…
Research instruments play significant roles in the construction of scientific knowledge, even though we have only acquired very limited knowledge about their lifecycles from quantitative studies. This paper aims to address this gap by quantitatively examining the citation contexts of an exemplary research instrument, the Diagnostic and Statistical Manual of Mental Disorders (DSM), in full-text psychological publications. We investigated the relationship between the citation contexts of the DSM and its status as a valid instrument being used and described by psychological researchers. We specif…
In this paper we present a first large-scale analysis of the relationship between Mendeley readership and citation counts with particular documents bibliographic characteristics. A data set of 1.3 million publications from different fields published in journals covered by the Web of Science (WoS) has been analyzed. This work reveals that document types that are often excluded from citation analysis due to their lower citation values, like editorial materials, letters, or news items, are strongly covered and saved in Mendeley, suggesting that Mendeley readership can reliably inform the analysis…
This study examines the correlational relationships between local journal authorship, local and external citation counts, full-text downloads, link-resolver clicks, and four global journal impact factor indices within an all-disciplines journal collection of 12,200 titles and six subject subsets at the University of Illinois at Urbana-Champaign (UIUC) Library. While earlier investigations of the relationships between usage (downloads) and citation metrics have been inconclusive, this study shows strong correlations in the all-disciplines set and most subject subsets. The normalized Eigenfactor…
We compare the visibility of Latin American and Caribbean (LAC) publications in the Core Collection indexes of the Web of Science (WoS)--Science Citation Index Expanded, Social Sciences Citation Index, and Arts & Humanities Citation Index--and the SciELO Citation Index (SciELO CI) which was integrated into the larger WoS platform in 2014. The purpose of this comparison is to contribute to our understanding of the communication of scientific knowledge produced in Latin America and the Caribbean, and to provide some reflections on the potential benefits of the articulation of regional indexing e…
This work looks in depth at several studies that have attempted to automate the process of citation importance classification based on the publications full text. We analyse a range of features that have been previously used in this task. Our experimental results confirm that the number of in text references are highly predictive of influence. Contrary to the work of Valenzuela et al. we find abstract similarity one of the most predictive features. Overall, we show that many of the features previously described in literature are not particularly predictive. Consequently, we discuss challenges …
This work analyzes the variation over time of the effect of geographic distance on knowledge flows. The flows are measured through the citations exchanged between scientific publications, including and excluding self-citations. To calculate geographic distances between citing and cited publication, each publication is associated with a "prevailing" territory, according to the authors' affiliations. We then apply a gravity model to account for the research size of the territories, in terms of cognitive proximity of citing-cited publications. The field of observation is the 2010-2017 world publi…
Studying the history of research fields by analyzing publication records and topical and/or keyword searches with a full Reference Publication Year Spectroscopy (RPYS) has been introduced as a powerful tool to identify the corresponding root publications. However, for a rather new and interdisciplinary research field like Solar Energy Meteorology (SEM), this method is not feasible to get a reasonably exhaustive publication set. Therefore we apply its variant RPYS-CO to all publications co-cited with one highly important marker paper, using the CRExplorer for plotting and inspecting the spectro…
The growing complexity of scientific challenges demands increasingly intense research collaboration, both domestic and international. The resulting trend affects not only the modes of producing new knowledge, but also the way it is disseminated within scientific communities. This paper analyses the relationship between the "degree of internationalization" of a country's scientific production and that of the relevant citing publications. The empirical analysis is based on 2010-2012 Italian publications. Findings show: i) the probability of being cited increases with the degree of internationali…
Citation impact is commonly assessed using direct, first-order citation relations. We consider here instead the indirect influence of publications on new publications via citations. We present a novel method to quantify the higher-order citation influence of publications, considering both direct, or first-order, and indirect, or higher-order citations. In particular, we are interested in higher-order citation influence at the level of disciplines. We apply this method to the whole Web of Science data at the level of disciplines. We find that a significant amount of influence -- 42% -- stems fr…
In course of the organization of Workshop III entitled "Cited References Analysis Using CRExplorer" at the International Conference of the International Society for Scientometrics and Informetrics (ISSI2021), we have prepared three reference publication year spectroscopy (RPYS) analyses: (i) papers published in Journal of Informetrics; (ii) papers regarding the topic altmetrics; and (iii) papers published by Ludo Waltman (we selected this researcher since he received the Derek de Solla Price Memorial Medal during the ISSI2021 conference). The first RPYS analysis has been presented live at the …
The article presents three advanced citation-based methods used to detect potential breakthrough papers among very highly cited papers. We approach the detection of such papers from three different perspectives in order to provide different typologies of breakthrough papers. In all three cases we use the classification of scientific publications developed at CWTS based on direct citation relationships. This classification establishes clusters of papers at three levels of aggregation. Papers are clustered based on their similar citation orientations and it is assumed that they are focused on si…
The effects of Open Access (OA) upon journal performance are investigated. The key research question holds: How does the citation impact and publication output of journals switching ('flipping') from non-OA to Gold-OA develop after their switch to Gold-OA? A review is given of the literature, with an emphasis on studies dealing with flipping journals. Two study sets with 119 and 100 flipping journals, derived from two different OA data sources (DOAJ and OAD), are compared with two control groups, one based on a standard bibliometric criterion, and a second controlling for a journal's national …
The prediction of the long-term impact of a scientific article is challenging task, addressed by the bibliometrician through resorting to a proxy whose reliability increases with the breadth of the citation window. In the national research assessment exercises using metrics the citation window is necessarily short, but in some cases is sufficient to advise the use of simple citations. For the Italian VQR 2011-2014, the choice was instead made to adopt a linear weighted combination of citations and journal metric percentiles, with weights differentiated by discipline and year. Given the strateg…
Empirical evidence shows that co-authored publications achieve higher visibility and impact. The aim of the current work is to test for the existence of a similar correlation for Italian publications. We also verify if such correlation differs: i) by subject category and macro-area; ii) by document type; iii) over the course of time. The results confirm world-level evidence, showing a consistent and significant linear growth in the citability of a publication with number of co-authors, in almost all subject categories. The effects are more remarkable in the fields of Social Sciences and Art & …
We describe a simple model of how a publication's citations change over time, based on pure-birth stochastic processes with a linear cumulative advantage effect. The model is applied to citation data from the Physical Review corpus provided by APS. Our model reveals that papers fall into three different clusters: papers that have rapid initial citations and ultimately high impact (fast-hi), fast to rise but quick to plateau (fast-flat), or late bloomers (slow-late), which may either never achieve many citations, or do so many years after publication. In "fast-hi" and "slow-late", there is a ri…
With the passage of more time from the original date of publication, the measure of the impact of scientific works using subsequent citation counts becomes more accurate. However the measurement of individual and organizational research productivity should ideally refer to a period with closing date just prior to the evaluation exercise. Therefore it is necessary to compromise between accuracy and timeliness. This work attempts to provide an order of magnitude for the error in measurement that occurs with decreasing the time lapse between date of publication and citation count. The analysis is…
In research policy, effective measures that lead to improvements in the generation of knowledge must be based on reliable methods of research assessment, but for many countries and institutions this is not the case. Publication and citation analyses can be used to estimate the part played by countries and institutions in the global progress of knowledge, but a concrete method of estimation is far from evident. The challenge arises because publications that report real progress of knowledge form an extremely low proportion of all publications; in most countries and institutions such contributio…
This work applies a new approach to measure knowledge flows. Assuming that citation linkages between articles imply a flow of knowledge from the cited to the citing authors, we investigate the geographic flows of scientific knowledge produced in Italy across its regions, at both overall and field level. Furthermore, we measure the the specialization indexes for outflows and inflows of knowledge by a given region. Findings show that larger regions in terms of research output are more likely net exporters of new knowledge. At the same time, we register a positive correlation between the share of…
In this work we ask whether and to what extent applying a predictor of publications' impact better than early citations, has an effect on the assessment of research performance of individual scientists. Specifically, we measure the total impact of Italian professors in the sciences and economics in a period of time, valuing their publications first by early citations and then by a weighted combination of early citations and impact factor of the hosting journal. As expected, scores and ranks by the two indicators show a very strong correlation, but there occur also significant shifts in many fi…
This study presents a large scale analysis of the distribution and presence of Mendeley readership scores over time and across disciplines. We study whether Mendeley readership scores (RS) can identify highly cited publications more effectively than journal citation scores (JCS). Web of Science (WoS) publications with DOIs published during the period 2004-2013 and across 5 major scientific fields have been analyzed. The main result of this study shows that readership scores are more effective (in terms of precision/recall values) than journal citation scores to identify highly cited publicatio…
Different scientific fields have different citation practices. Citation-based bibliometric indicators need to normalize for such differences between fields in order to allow for meaningful between-field comparisons of citation impact. Traditionally, normalization for field differences has usually been done based on a field classification system. In this approach, each publication belongs to one or more fields and the citation impact of a publication is calculated relative to the other publications in the same field. Recently, the idea of source normalization was introduced, which offers an alt…
Academic publications have been evaluated in terms of their impact on research communities based on many metrics, such as the number of citations. On the other hand, the impact of academic publications on industry has been rarely studied. This paper investigates how academic publications contribute to software development by analyzing publication citations in source code comments in open source software repositories. We propose an automated approach for detecting academic publications based on Named Entity Recognition, and achieve 0.90 in $F_1$ as detection accuracy. We conduct a large-scale s…
In this study, I analyze the DBLP bibliographic database to study role of single author publications in the computer science literature between 1940 and 2019. I examine the demographics and reception by computing the population fraction, citation statistics, and PageRank scores of single author publications over the years. Both the population fraction and reception have been continuously declining since the 1940s. The overall decaying trend of single author publications is qualitatively consistent with those observed in other scientific disciplines, though the diminution is taking place severa…
We present a large-scale analysis of the accuracy of citation data in the Web of Science and Scopus databases. The analysis is based on citations given in publications in Elsevier journals. We reveal significant data quality problems for both databases. Missing and incorrect references are important problems in Web of Science. Duplicate publications are a serious problem in Scopus.
World University rankings have become well-established tools that students, university managers and policy makers read and use. Each ranking claims to have a unique methodology capable of measuring the 'quality' of universities. The purpose of this paper is to analyze to which extent these different rankings measure the same phenomenon and what it is that they are measuring. For this, we selected a total of seven world-university rankings and performed a principal component analysis. After ensuring that despite their methodological differences, they all come together to a single component, we …
Is more always better? We address this question in the context of bibliometric indices that aim to assess the scientific impact of individual researchers by counting their number of highly cited publications. We propose a simple model in which the number of citations of a publication depends not only on the scientific impact of the publication but also on other 'random' factors. Our model indicates that more need not always be better. It turns out that the most influential researchers may have a systematically lower performance, in terms of highly cited publications, than some of their less in…
Digital Science's Dimensions is envisaged as a next-generation research and discovery platform for a better and more efficient access to cross-referenced scholarly publications, grants, patents, and clinical trials. As a new addition to the growing open citation resources, it offers opportunities that may benefit a wide variety of stakeholders of scientific publications from researchers, policy makers, and the general public. In this article, we explore and demonstrate some of the practical potentials in terms of cascading citation expansions. Given a set of publications, the cascading citatio…
College of Computing and Informatics, Drexel University（德雷塞尔大学计算与信息学院）
Researchers may describe different aspects of past scientific publications in their publications and the descriptions may keep changing in the evolution of science. The diverse and changing descriptions (i.e., citation context) on a publication characterize the impact and contributions of the past publication. In this article, we aim to provide an approach to understanding the changing and complex roles of a publication characterized by its citation context. We described a method to represent the publications' dynamic roles in science community in different periods as a sequence of vectors by …
We analyze the citation distributions of all papers published in Physical Review journals between 1985 and 2009. The average number of citations received by papers published in a given year and in a given field is computed. Large variations are found, showing that it is not fair to compare citation numbers across fields and years. However, when a rescaling procedure by the average is used, it is possible to compare impartially articles across years and fields. We make the rescaling factors available for use by the readers. We also show that rescaling citation numbers by the number of publicati…
The two most used citation impact indicators in the assessment of scientific journals are, nowadays, the impact factor and the h-index. However, both indicators are not field normalized (vary heavily depending on the scientific category) which makes them incomparable between categories. Furthermore, the impact factor is not robust to the presence of articles with a large number of citations, while the h-index depends on the journal size. These limitations are very important when comparing journals of different sizes and categories. An alternative citation impact indicator is the percentage of …
