Eigenfactor.org, a journal evaluation tool which uses an iterative algorithm to weight citations (similar to the PageRank algorithm used for Google) has been proposed as a more valid method for calculating the impact of journals. The purpose of this brief communication is to investigate whether the principle of repeated improvement provides different rankings of journals than does a simple unweighted citation count (the method used by ISI).
The Eigenfactor is a journal metric, which was developed by Bergstrom and his colleagues at the University of Washington. They invented the Eigenfactor as a response to the criticism against the use of simple citation counts. The Eigenfactor makes use of the network structure of citations, i.e. citations between journals, and establishes the importance, influence or impact of a journal based on its location in a network of journals. The importance is defined based on the number of citations between journals. As such, the Eigenfactor algorithm is based on Eigenvector centrality. While journal-b…
The Eigenfactor Metrics provide an alternative way of evaluating scholarly journals based on an iterative ranking procedure analogous to Google's PageRank algorithm. These metrics have recently been adopted by Thomson-Reuters and are listed alongside the Impact Factor in the Journal Citation Reports. But do these metrics differ sufficiently so as to be a useful addition to the bibliometric toolbox? Davis (2008) has argued otherwise, based on his finding of a 0.95 correlation coefficient between Eigenfactor score and Total Citations for a sample of journals in the field of medicine. This conclu…
Microsoft Research hosted the 2016 WSDM Cup Challenge based on the Microsoft Academic Graph. The goal was to provide static rankings for the articles that make up the graph, with the rankings to be evaluated against those of human judges. While the Microsoft Academic Graph provided metadata about many aspects of each scholarly document, we focused more narrowly on citation data and used this contest as an opportunity to test the Article Level Eigenfactor (ALEF), a novel citation-based ranking algorithm, and evaluate its performance against competing algorithms that drew upon multiple facets of…
We present a theoretical and empirical analysis of a number of bibliometric indicators of journal performance. We focus on three indicators in particular, namely the Eigenfactor indicator, the audience factor, and the influence weight indicator. Our main finding is that the last two indicators can be regarded as a kind of special cases of the first indicator. We also find that the three indicators can be nicely characterized in terms of two properties. We refer to these properties as the property of insensitivity to field differences and the property of insensitivity to insignificant journals.…
The journal structure in the China Scientific and Technical Papers and Citations Database (CSTPCD) is analysed from three perspectives: the database level, the specialty level and the institutional level (i.e., university journals versus journals issued by the Chinese Academy of Sciences). The results are compared with those for (Chinese) journals included in the Science Citation Index. The frequency of journal-journal citation relations in the CSTPCD is an order of magnitude lower than in the SCI. Chinese journals, especially high-quality journals, prefer to cite international journals rather…
A number of journal classification systems have been developed in bibliometrics since the launch of the Citation Indices by the Institute of Scientific Information (ISI) in the 1960s. These systems are used to normalize citation counts with respect to field-specific citation patterns. The best known system is the so-called "Web-of-Science Subject Categories" (WCs). In other systems papers are classified by algorithmic solutions. Using the Journal Citation Reports 2014 of the Science Citation Index and the Social Science Citation Index (n of journals = 11,149), we examine options for developing…
The journal impact factor is not comparable among fields of science and social science because of systematic differences in publication and citation behaviour across disciplines. In this work, a source normalization of the journal impact factor is proposed. We use the aggregate impact factor of the citing journals as a measure of the citation potential in the journal topic, and we employ this citation potential in the normalization of the journal impact factor to make it comparable between scientific fields. An empirical application comparing some impact indicators with our topic normalized im…
This study examines the correlational relationships between local journal authorship, local and external citation counts, full-text downloads, link-resolver clicks, and four global journal impact factor indices within an all-disciplines journal collection of 12,200 titles and six subject subsets at the University of Illinois at Urbana-Champaign (UIUC) Library. While earlier investigations of the relationships between usage (downloads) and citation metrics have been inconclusive, this study shows strong correlations in the all-disciplines set and most subject subsets. The normalized Eigenfactor…
We compare the network of aggregated journal-journal citation relations provided by the Journal Citation Reports (JCR) 2012 of the Science and Social Science Citation Indexes (SCI and SSCI) with similar data based on Scopus 2012. First, global maps were developed for the two sets separately; sets of documents can then be compared using overlays to both maps. Using fuzzy-string matching and ISSN numbers, we were able to match 10,524 journal names between the two sets; that is, 96.4% of the 10,936 journals contained in JCR or 51.2% of the 20,554 journals covered by Scopus. Network analysis was t…
Aggregated journal-journal citation networks based on the Journal Citation Reports 2004 of the Science Citation Index (5968 journals) and the Social Science Citation Index (1712 journals) are made accessible from the perspective of any of these journals. The user is thus able to analyze the citation environment in terms of links and graphs. Furthermore, the local impact of a journal is defined as its share of the total citations in the specific journal's citation environments; the vertical size of the nodes is varied proportionally to this citation impact. The horizontal size of each node can …
The launch of Google Scholar Metrics as a tool for assessing scientific journals may be serious competition for Thomson Reuters Journal Citation Reports, and for Scopus powered Scimago Journal Rank. A review of these bibliometric journal evaluation products is performed. We compare their main characteristics from different approaches: coverage, indexing policies, search and visualization, bibliometric indicators, results analysis options, economic cost and differences in their ranking of journals. Despite its shortcomings, Google Scholar Metrics is a helpful tool for authors and editors in ide…
Evaluating and comparing the academic performance of a journal, a researcher or a single paper has long remained a critical, necessary but also controversial issue. Most of existing metrics invalidate comparison across different fields of science or even between different types of papers in the same field. This paper proposes a new metric, called return on citation (ROC), which is simply a citation ratio but applies to evaluating the paper, the journal and the researcher in a consistent way, allowing comparison across different fields of science and between different types of papers and discou…
Davis (2008) analyzes citations from 2004-2007 in 11 biomedical journals. 15% of authors paid to make them Open Access (OA). The outcome is a significant OA citation Advantage, but a small one (21%). The author infers that the OA advantage has been shrinking yearly, but the data suggest the opposite. Further analyses are necessary: (1) Not just author-choice (paid) OA but Free OA self-archiving needs to be taken into account rather than being counted as non-OA. (2) proportion of OA articles per journal per year needs to be reported and taken into account. (3) The Journal Impact Factor and the …
We generated networks of journal relationships from citation and download data, and determined journal impact rankings from these networks using a set of social network centrality metrics. The resulting journal impact rankings were compared to the ISI IF. Results indicate that, although social network metrics and ISI IF rankings deviate moderately for citation-based journal networks, they differ considerably for journal networks derived from download data. We believe the results represent a unique aspect of general journal impact that is not captured by the ISI IF. These results furthermore ra…
A statistical analysis of full text downloads of articles in Elseviers ScienceDirect covering all disciplines reveals large differences in download frequencies, their skewness, and their correlation with Scopus-based citation counts, between disciplines, journals, and document types. Download counts tend to be two orders of magnitude higher and less skewedly distributed than citations. A mathematical model based on the sum of two exponentials does not adequately capture monthly download counts. The degree of correlation at the article level within a journal is similar to that at the journal le…
Evaluating the quality of academic journal is becoming increasing important within the context of research performance evaluation. Traditionally, journals have been ranked by peer review lists such as that of the Association of Business Schools (UK) or though their journal impact factor (JIF). However, several new indicators have been developed, such as the h-index, SJR, SNIP and the Eigenfactor which take into account different factors and therefore have their own particular biases. In this paper we evaluate these metrics both theoretically and also through an empirical study of a large set o…
Based on the citation data of journals covered by the China Scientific and Technical Papers and Citations Database (CSTPCD), we obtained aggregated journal-journal citation environments by applying routines developed specifically for this purpose. Local citation impact of journals is defined as the share of the total citations in a local citation environment, which is expressed as a ratio and can be visualized by the size of the nodes. The vertical size of the nodes varies proportionally to a journal's total citation share, while the horizontal size of the nodes is used to provide citation inf…
The possibilities of using the Arts & Humanities Citation Index (A&HCI) for journal mapping have not been sufficiently recognized because of the absence of a Journal Citations Report (JCR) for this database. A quasi-JCR for the A&HCI (2008) was constructed from the data contained in the Web-of-Science and is used for the evaluation of two journals as examples: Leonardo and Art Journal. The maps on the basis of the aggregated journal-journal citations within this domain can be compared with maps including references to journals in the Science Citation Index and Social Science Citation Index. Ar…
Can change in citation patterns among journals be used as an indicator of structural change in the organization of the sciences? Aggregated journal-journal citations for 1999 are compared with similar data in the Journal Citation Reports 1998 of the Science Citation Index. In addition to indicating local change, probabilistic entropy measures enable us to analyze changes in distributions at different levels of aggregation. The results of various statistics are discussed and compared by elaborating the journal-journal mappings. The relevance of this indicator for science and technology policies…
Using three-year moving averages of the complete Journal Citation Reports 1994-2016 of the Science Citation Index and the Social Sciences Citation Index (combined), we analyze links between citing and cited journals in terms of (1) whether discontinuities among the networks of consecutive years have occurred; (2) are these discontinuities relatively isolated or networked? (3) Can these discontinuities be used as indicators of novelty, change, and innovation in the sciences? We examine each of the N2 links among the N journals across the years. We find power-laws for the top 10,000 instances of…
In this paper we present "citation success index", a metric for comparing the citation capacity of pairs of journals. Citation success index is the probability that a random paper in one journal has more citations than a random paper in another journal (50% means the two journals do equally well). Unlike the journal impact factor (IF), the citation success index depends on the broadness and the shape of citation distributions. Also, it is insensitive to sporadic highly-cited papers that skew the IF. Nevertheless, we show, based on 16,000 journals containing ~2.4 million articles, that the cita…
Using three years of the Journal Citation Reports (2011, 2012, and 2013), indicators of transitions in 2012 (between 2011 and 2013) are studied using methodologies based on entropy statistics. Changes can be indicated at the level of journals using the margin totals of entropy production along the row or column vectors, but also at the level of links among journals by importing the transition matrices into network analysis and visualization programs (and using community-finding algorithms). Seventy-four journals are flagged in terms of discontinuous changes in their citations; but 3,114 journa…
The Journal Impact Factor (JIF) is linearly sensitive to self-citations because each self-citation adds to the numerator, whereas the denominator is not affected. Pinski & Narin (1976) derived the Influence Weight (IW) as an alternative to Garfield's JIF. Whereas the JIF is based on raw citation counts normalized by the number of publications, IWs are based on the eigenvectors in the matrix of aggregated journal-journal citations without a reference to size: the cited and citing sides are combined by a matrix approach. IWs emerge as a vector after recursive iteration of the normalized matrix. …
Journal Impact Factor is a popular metric for determining the quality of a journal in academia. The number of citations received by a journal is a crucial factor in determining the impact factor, which may be misused in multiple ways. Therefore, it is crucial to detect citation anomalies for further identifying manipulation and inflation of impact factor. Citation network models the citation relationship between journals in terms of a directed graph. Detecting anomalies in the citation network is a challenging task which has several applications in spotting citation cartels and citation stack …
We analysed 580 articles (original articles only) published in Medical Journal of Malaysia between 2004 and 2008, the resources referenced by the articles and the citations and impact received. Our aim was to examine article and author productivity, the age of references used and impact of the journal. Publication data was obtained from MyAIS database and Google Scholar provided the citation data. From the 580 articles analyzed, contributors mainly come from the hospitals, universities and clinics. Contributions from foreign authors are low. The useful lives of references cited were between 3 …
Microsoft Academic is a free citation index that allows large scale data collection. This combination makes it useful for scientometric research. Previous studies have found that its citation counts tend to be slightly larger than those of Scopus but smaller than Google Scholar, with disciplinary variations. This study reports the largest and most systematic analysis so far, of 172,752 articles in 29 large journals chosen from different specialisms. From Scopus citation counts, Microsoft Academic citation counts and Mendeley reader counts for articles published 2007-2017, Microsoft Academic fo…
Using "Analyze Results" at the Web of Science, one can directly generate overlays onto global journal maps of science. The maps are based on the 10,000+ journals contained in the Journal Citation Reports (JCR) of the Science and Social Science Citation Indices (2011). The disciplinary diversity of the retrieval is measured in terms of Rao-Stirling's "quadratic entropy." Since this indicator of interdisciplinarity is normalized between zero and one, the interdisciplinarity can be compared among document sets and across years, cited or citing. The colors used for the overlays are based on Blonde…
Dyads of journals related by citations can agglomerate into specialties through the mechanism of triadic closure. Using the Journal Citation Reports 2011, 2012, and 2013, we analyze triad formation as indicators of integration (specialty growth) and disintegration (restructuring). The strongest integration is found among the large journals that report on studies in different scientific specialties, such as PLoS ONE, Nature Communications, Nature, and Science. This tendency towards large-scale integration has not yet stabilized. Using the Islands algorithm, we also distinguish 51 local maxima o…
Scientific journals are an important choice of publication venue for most authors. Publishing in prestigious journal plays a decisive role for authors in hiring and promotions. In last decade, citation pressure has become intact for all scientific entities more than ever before. Unethical publication practices has started to manipulate widely used performance metric such as "impact factor" for journals and citation based indices for authors. This threatens the integrity of scientific quality and takes away deserved credit of legitimate authors and their authentic publications. In this paper we…
Single journal studies are reviewed and measures used in the studies are highlighted. The following quantitative measures are used to study 272 articles published in Malaysian Journal of Computer Science, (1) the article productivity of the journal from 1985 to 2007, (2) the observed and expected authorship productivity tested using Lotka's Law of author productivity, identification and listing of core authors; (3) the authorship, co-authorship pattern by authors' country of origin and institutional affiliations; (4) the subject areas of research; (5) the citation analysis of resources referen…
Evaluation of journals for quality is one of the dominant themes of bibliometrics since journals are the primary venue of vetting and distribution of scholarship. There are many criticisms of quantifying journal impact with bibliometrics including disciplinary differences among journals, what source materials are used, time windows for the inclusion of works to measure, and skewness of citation distributions (Lariviere & Sugimoto, 2019). However, despite various attempts to remediate these in newly proposed indicators such as SJR, SNIP, and Eigenfactor (Walters, 2017) indicators still remain b…
Articles in high-impact journals are, on average, more frequently cited. But are they cited more often because those articles are somehow more "citable"? Or are they cited more often simply because they are published in a high-impact journal? Although some evidence suggests the latter, the causal relationship is not clear. We here compare citations of preprints to citations of the published version to uncover the causal mechanism. We build on an earlier model of citation dynamics to infer the causal effect of journals on citations. We find that high-impact journals select articles that tend to…
It is shown that the calculation of scientometric indicators of the scientist and also the scientific journal continues to be an actual problem nowadays. It is revealed that the leading scientometric databases have the capabilities of automated metadata collection from the scientific journal website by the use of specialized electronic document management systems, in particular Open Journal Systems. It is established that Open Journal Systems successfully exports metadata about an article from scientific journals to scientometric databases Scopus, Web of Science and Google Scholar. However, th…
The two most used citation impact indicators in the assessment of scientific journals are, nowadays, the impact factor and the h-index. However, both indicators are not field normalized (vary heavily depending on the scientific category) which makes them incomparable between categories. Furthermore, the impact factor is not robust to the presence of articles with a large number of citations, while the h-index depends on the journal size. These limitations are very important when comparing journals of different sizes and categories. An alternative citation impact indicator is the percentage of …
The aggregated citation relations among journals included in the Science Citation Index provide us with a huge matrix which can be analyzed in various ways. Using principal component analysis or factor analysis, the factor scores can be used as indicators of the position of the cited journals in the citing dimensions of the database. Unrotated factor scores are exact, and the extraction of principal components can be made stepwise since the principal components are independent. Rotation may be needed for the designation, but in the rotated solution a model is assumed. This assumption can be le…
In December 2016, after several years of development, Elsevier launched a set of transparent, comprehensive, current, and freely-available journal citation metrics called CiteScore metrics. Most of the CiteScore metrics are static, annual indicators calculated from the dynamic Scopus citation index. In the spirit of recent public statements on the responsible use of metrics, we outline the desirable characteristics of journal citation metrics, discuss how we decided on the cited and citing publications years and document types to be used for CiteScore metrics, and detail the precise method of …
In this study, the academic status of users of scientific publications in Mendeley is explored in order to analyse the usage pattern of Mendeley users in terms of subject fields, citation and readership impact. The main focus of this study is on studying the filtering capacity of Mendeley readership counts compared to journal citation scores in detecting highly cited WoS publications. Main finding suggests a faster reception of Mendeley readerships as compared to citations across 5 major field of science. The higher correlations of scientific users with citations indicate the similarity betwee…
Rankings of scholarly journals based on citation data are often met with skepticism by the scientific community. Part of the skepticism is due to disparity between the common perception of journals' prestige and their ranking based on citation counts. A more serious concern is the inappropriate use of journal rankings to evaluate the scientific influence of authors. This paper focuses on analysis of the table of cross-citations among a selection of Statistics journals. Data are collected from the Web of Science database published by Thomson Reuters. Our results suggest that modelling the excha…
A citation-based indicator for interdisciplinarity has been missing hitherto among the set of available journal indicators. In this study, we investigate network indicators (betweenness centrality), journal indicators (Shannon entropy, the Gini coefficient), and more recently proposed Rao-Stirling measures for "interdisciplinarity." The latter index combines the statistics of both citation distributions of journals (vector-based) and distances in citation networks among journals (matrix-based). The effects of various normalizations are specified and measured using the matrix of 8,207 journals …
The aggregated journal-journal citation matrix derived from the Journal Citation Reports 2001 can be decomposed into a unique subject classification by using the graph-analytical algorithm of bi-connected components. This technique was recently incorporated in software tools for social network analysis. The matrix can be assessed in terms of its decomposability using articulation points which indicate overlap between the components. The articulation points of this set did not exhibit a next-order network of 'general science' journals. However, the clusters differ in size and in terms of the in…
Ramanujacharyulu's (1964) Power-Weakness Ratio (PWR) measures impact by recursively multiplying the citation matrix by itself until convergence is reached in both the cited and citing dimensions; the quotient of these values is defined as PWR, whereby "cited" is considered as power and "citing" as weakness. Analytically, PWR is an attractive candidate for measuring journal impact because of its symmetrical handling of the rows and columns in the asymmetrical citation matrix, its recursive algorithm, and its mathematical elegance. In this study, PWR is discussed and critically assessed in relat…
This paper explores a new indicator of journal citation impact, denoted as source normalized impact per paper (SNIP). It measures a journal's contextual citation impact, taking into account characteristics of its properly defined subject field, especially the frequency at which authors cite other papers in their reference lists, the rapidity of maturing of citation impact, and the extent to which a database used for the assessment covers the field's literature. It further develops Eugene Garfield's notions of a field's 'citation potential' defined as the average length of references lists in a…
A key feature of questionable journals is a lack of adequate peer review of their articles. Content of thus unknown quality may be utilised by unsuspecting practitioners or incorporated into peer-reviewed research, becoming legitimised. It is therefore necessary to examine the citation patterns of articles in questionable journals to understand the impact and reach of research in questionable journals. Similar research has tended to focus on authors from low- and middle-income countries. As such, this study investigates the profile and impact of research in questionable journals by authors in …
This paper investigates the Twitter interaction patterns of journals from the Science Citation Index (SCI) of Master Journal List (MJL). A total of 953,253 tweets extracted from 857 journal accounts, were analyzed in this study. Findings indicate that SCI journals interacted more with each other but much less with journals from other citation indices. The network structure of the communication graph resembled a tight crowd network, with Nature journals playing a major part. Information sources such as news portals and scientific organizations were mentioned more in tweets, than academic journa…
The purpose of citation indexes and metrics is intended to be a measure for scientific innovation and quality for researchers, journals, and institutions. However, those metrics are often prone to abuse and manipulation by excessive and unethical self-citations induced by authors, reviewers, editors, or journals. Identifying whether there are or not legitimate reasons for self-citations is normally determined during the review process, where the participating parts may have intrinsic incentives, rendering the legitimacy of self-citations, after publication, questionable. In this paper, we cond…
Microsoft Academic is a free academic search engine and citation index that is similar to Google Scholar but can be automatically queried. Its data is potentially useful for bibliometric analysis if it is possible to search effectively for individual journal articles. This article compares different methods to find journal articles in its index by searching for a combination of title, authors, publication year and journal name and uses the results for the widest published correlation analysis of Microsoft Academic citation counts for journal articles so far. Based on 126,312 articles from 323 …
Two indicators rule them all: Mean and standard deviation used to calculate other journal indicators based on log-normal distribution of citation counts
We demonstrate conclusively that there is no "Open Access Advantage" for papers from the Astrophysical Journal. The two to one citation advantage enjoyed by papers deposited in the arXiv e-print server is due entirely to the nature and timing of the deposited papers. This may have implications for other disciplines.
Ageing of publications, percentage of self-citations, and impact vary from journal to journal within fields of science. The assumption that citation and publication practices are homogenous within specialties and fields of science is invalid. Furthermore, the delineation of fields and among specialties is fuzzy. Institutional units of analysis and persons may move between fields or span different specialties. The match between the citation index and institutional profiles varies among institutional units and nations. The respective matches may heavily affect the representation of the units. No…
This research analyzes a "who cites whom" matrix in terms of aggregated, journal-journal citations to determine the location of communication studies on the academic spectrum. Using the Journal of Communication as the seed journal, the 2006 data in the Journal Citation Reports are used to map communication studies. The results show that social and experimental psychology journals are the most frequently used sources of information in this field. In addition, several journals devoted to the use and effects of media and advertising are weakly integrated into the larger communication research com…
We analyse the large-scale structure of the journal citation network built from information contained in the Thomson-Reuters Journal Citation Reports. To this end, we take advantage of the network science paraphernalia and explore network properties like density, percolation robustness, average and largest node distances, reciprocity, incoming and outgoing degree distributions, as well as assortative mixing by node degrees. We discover that the journal citation network is a dense, robust, small, and reciprocal world. Furthermore, in and out node degree distributions display long-tails, with fe…
Questions of definition and measurement continue to constrain a consensus on the measurement of interdisciplinarity. Using Rao-Stirling (RS) Diversity produces sometimes anomalous results. We argue that these unexpected outcomes can be related to the use of "dual-concept diversity" which combines "variety" and "balance" in the definitions (ex ante). We propose to modify RS Diversity into a new indicator (DIV) which operationalizes variety, balance, and disparity independently and then combines them ex post. "Balance" can be measured using the Gini coefficient. We apply DIV to the aggregated ci…
The ability to predict the long-term impact of a scientific article soon after its publication is of great value towards accurate assessment of research performance. In this work we test the hypothesis that good predictions of long-term citation counts can be obtained through a combination of a publication's early citations and the impact factor of the hosting journal. The test is performed on a corpus of 123,128 WoS publications authored by Italian scientists, using linear regression models. The average accuracy of the prediction is good for citation time windows above two years, decreases fo…
