Many factors could affect the number of citations to a paper. Citations have an important role in research policy and in measuring the excellence of research and researchers. This work is the first study in software engineering (SE) to assess multiple factors affecting the number of citations to SE papers. We use (a) negative binomial regression and (b) quantile regression to study arithmetic mean and median expected citations of a paper. Our dataset includes all the 25,113 papers which have been published in a set of 16 main SE journals, between 1970 and 2018. Our results indicate that publication venue, author team's past citations, paper length, the number of references, and the recency of references are the most influential factors on the number of citations to SE papers. From our empirical findings, we present several implications and advice to researchers for getting higher citations on their papers, which are in addition to the obvious case of conducting high-quality technical research, e.g. (1) Aim for high-profile venues, (2) Build a high-quality author team with highly cited past papers, and (3) Aim for high-quality work that has comprehensive content (thus longer paper length and reference list).
The journal structure in the China Scientific and Technical Papers and Citations Database (CSTPCD) is analysed from three perspectives: the database level, the specialty level and the institutional level (i.e., university journals versus journals issued by the Chinese Academy of Sciences). The results are compared with those for (Chinese) journals included in the Science Citation Index. The frequency of journal-journal citation relations in the CSTPCD is an order of magnitude lower than in the SCI. Chinese journals, especially high-quality journals, prefer to cite international journals rather…
A number of journal classification systems have been developed in bibliometrics since the launch of the Citation Indices by the Institute of Scientific Information (ISI) in the 1960s. These systems are used to normalize citation counts with respect to field-specific citation patterns. The best known system is the so-called "Web-of-Science Subject Categories" (WCs). In other systems papers are classified by algorithmic solutions. Using the Journal Citation Reports 2014 of the Science Citation Index and the Social Science Citation Index (n of journals = 11,149), we examine options for developing…
The journal impact factor is not comparable among fields of science and social science because of systematic differences in publication and citation behaviour across disciplines. In this work, a source normalization of the journal impact factor is proposed. We use the aggregate impact factor of the citing journals as a measure of the citation potential in the journal topic, and we employ this citation potential in the normalization of the journal impact factor to make it comparable between scientific fields. An empirical application comparing some impact indicators with our topic normalized im…
Aggregated journal-journal citation networks based on the Journal Citation Reports 2004 of the Science Citation Index (5968 journals) and the Social Science Citation Index (1712 journals) are made accessible from the perspective of any of these journals. The user is thus able to analyze the citation environment in terms of links and graphs. Furthermore, the local impact of a journal is defined as its share of the total citations in the specific journal's citation environments; the vertical size of the nodes is varied proportionally to this citation impact. The horizontal size of each node can …
Based on the citation data of journals covered by the China Scientific and Technical Papers and Citations Database (CSTPCD), we obtained aggregated journal-journal citation environments by applying routines developed specifically for this purpose. Local citation impact of journals is defined as the share of the total citations in a local citation environment, which is expressed as a ratio and can be visualized by the size of the nodes. The vertical size of the nodes varies proportionally to a journal's total citation share, while the horizontal size of the nodes is used to provide citation inf…
We compare the network of aggregated journal-journal citation relations provided by the Journal Citation Reports (JCR) 2012 of the Science and Social Science Citation Indexes (SCI and SSCI) with similar data based on Scopus 2012. First, global maps were developed for the two sets separately; sets of documents can then be compared using overlays to both maps. Using fuzzy-string matching and ISSN numbers, we were able to match 10,524 journal names between the two sets; that is, 96.4% of the 10,936 journals contained in JCR or 51.2% of the 20,554 journals covered by Scopus. Network analysis was t…
The possibilities of using the Arts & Humanities Citation Index (A&HCI) for journal mapping have not been sufficiently recognized because of the absence of a Journal Citations Report (JCR) for this database. A quasi-JCR for the A&HCI (2008) was constructed from the data contained in the Web-of-Science and is used for the evaluation of two journals as examples: Leonardo and Art Journal. The maps on the basis of the aggregated journal-journal citations within this domain can be compared with maps including references to journals in the Science Citation Index and Social Science Citation Index. Ar…
Can change in citation patterns among journals be used as an indicator of structural change in the organization of the sciences? Aggregated journal-journal citations for 1999 are compared with similar data in the Journal Citation Reports 1998 of the Science Citation Index. In addition to indicating local change, probabilistic entropy measures enable us to analyze changes in distributions at different levels of aggregation. The results of various statistics are discussed and compared by elaborating the journal-journal mappings. The relevance of this indicator for science and technology policies…
In this paper we present "citation success index", a metric for comparing the citation capacity of pairs of journals. Citation success index is the probability that a random paper in one journal has more citations than a random paper in another journal (50% means the two journals do equally well). Unlike the journal impact factor (IF), the citation success index depends on the broadness and the shape of citation distributions. Also, it is insensitive to sporadic highly-cited papers that skew the IF. Nevertheless, we show, based on 16,000 journals containing ~2.4 million articles, that the cita…
Journal Impact Factor is a popular metric for determining the quality of a journal in academia. The number of citations received by a journal is a crucial factor in determining the impact factor, which may be misused in multiple ways. Therefore, it is crucial to detect citation anomalies for further identifying manipulation and inflation of impact factor. Citation network models the citation relationship between journals in terms of a directed graph. Detecting anomalies in the citation network is a challenging task which has several applications in spotting citation cartels and citation stack …
The launch of Google Scholar Metrics as a tool for assessing scientific journals may be serious competition for Thomson Reuters Journal Citation Reports, and for Scopus powered Scimago Journal Rank. A review of these bibliometric journal evaluation products is performed. We compare their main characteristics from different approaches: coverage, indexing policies, search and visualization, bibliometric indicators, results analysis options, economic cost and differences in their ranking of journals. Despite its shortcomings, Google Scholar Metrics is a helpful tool for authors and editors in ide…
Evaluating and comparing the academic performance of a journal, a researcher or a single paper has long remained a critical, necessary but also controversial issue. Most of existing metrics invalidate comparison across different fields of science or even between different types of papers in the same field. This paper proposes a new metric, called return on citation (ROC), which is simply a citation ratio but applies to evaluating the paper, the journal and the researcher in a consistent way, allowing comparison across different fields of science and between different types of papers and discou…
Evaluation of journals for quality is one of the dominant themes of bibliometrics since journals are the primary venue of vetting and distribution of scholarship. There are many criticisms of quantifying journal impact with bibliometrics including disciplinary differences among journals, what source materials are used, time windows for the inclusion of works to measure, and skewness of citation distributions (Lariviere & Sugimoto, 2019). However, despite various attempts to remediate these in newly proposed indicators such as SJR, SNIP, and Eigenfactor (Walters, 2017) indicators still remain b…
We generated networks of journal relationships from citation and download data, and determined journal impact rankings from these networks using a set of social network centrality metrics. The resulting journal impact rankings were compared to the ISI IF. Results indicate that, although social network metrics and ISI IF rankings deviate moderately for citation-based journal networks, they differ considerably for journal networks derived from download data. We believe the results represent a unique aspect of general journal impact that is not captured by the ISI IF. These results furthermore ra…
A statistical analysis of full text downloads of articles in Elseviers ScienceDirect covering all disciplines reveals large differences in download frequencies, their skewness, and their correlation with Scopus-based citation counts, between disciplines, journals, and document types. Download counts tend to be two orders of magnitude higher and less skewedly distributed than citations. A mathematical model based on the sum of two exponentials does not adequately capture monthly download counts. The degree of correlation at the article level within a journal is similar to that at the journal le…
The aggregated citation relations among journals included in the Science Citation Index provide us with a huge matrix which can be analyzed in various ways. Using principal component analysis or factor analysis, the factor scores can be used as indicators of the position of the cited journals in the citing dimensions of the database. Unrotated factor scores are exact, and the extraction of principal components can be made stepwise since the principal components are independent. Rotation may be needed for the designation, but in the rotated solution a model is assumed. This assumption can be le…
Articles in high-impact journals are, on average, more frequently cited. But are they cited more often because those articles are somehow more "citable"? Or are they cited more often simply because they are published in a high-impact journal? Although some evidence suggests the latter, the causal relationship is not clear. We here compare citations of preprints to citations of the published version to uncover the causal mechanism. We build on an earlier model of citation dynamics to infer the causal effect of journals on citations. We find that high-impact journals select articles that tend to…
Using three-year moving averages of the complete Journal Citation Reports 1994-2016 of the Science Citation Index and the Social Sciences Citation Index (combined), we analyze links between citing and cited journals in terms of (1) whether discontinuities among the networks of consecutive years have occurred; (2) are these discontinuities relatively isolated or networked? (3) Can these discontinuities be used as indicators of novelty, change, and innovation in the sciences? We examine each of the N2 links among the N journals across the years. We find power-laws for the top 10,000 instances of…
Using three years of the Journal Citation Reports (2011, 2012, and 2013), indicators of transitions in 2012 (between 2011 and 2013) are studied using methodologies based on entropy statistics. Changes can be indicated at the level of journals using the margin totals of entropy production along the row or column vectors, but also at the level of links among journals by importing the transition matrices into network analysis and visualization programs (and using community-finding algorithms). Seventy-four journals are flagged in terms of discontinuous changes in their citations; but 3,114 journa…
The Journal Impact Factor (JIF) is linearly sensitive to self-citations because each self-citation adds to the numerator, whereas the denominator is not affected. Pinski & Narin (1976) derived the Influence Weight (IW) as an alternative to Garfield's JIF. Whereas the JIF is based on raw citation counts normalized by the number of publications, IWs are based on the eigenvectors in the matrix of aggregated journal-journal citations without a reference to size: the cited and citing sides are combined by a matrix approach. IWs emerge as a vector after recursive iteration of the normalized matrix. …
We analysed 580 articles (original articles only) published in Medical Journal of Malaysia between 2004 and 2008, the resources referenced by the articles and the citations and impact received. Our aim was to examine article and author productivity, the age of references used and impact of the journal. Publication data was obtained from MyAIS database and Google Scholar provided the citation data. From the 580 articles analyzed, contributors mainly come from the hospitals, universities and clinics. Contributions from foreign authors are low. The useful lives of references cited were between 3 …
Microsoft Academic is a free citation index that allows large scale data collection. This combination makes it useful for scientometric research. Previous studies have found that its citation counts tend to be slightly larger than those of Scopus but smaller than Google Scholar, with disciplinary variations. This study reports the largest and most systematic analysis so far, of 172,752 articles in 29 large journals chosen from different specialisms. From Scopus citation counts, Microsoft Academic citation counts and Mendeley reader counts for articles published 2007-2017, Microsoft Academic fo…
Using "Analyze Results" at the Web of Science, one can directly generate overlays onto global journal maps of science. The maps are based on the 10,000+ journals contained in the Journal Citation Reports (JCR) of the Science and Social Science Citation Indices (2011). The disciplinary diversity of the retrieval is measured in terms of Rao-Stirling's "quadratic entropy." Since this indicator of interdisciplinarity is normalized between zero and one, the interdisciplinarity can be compared among document sets and across years, cited or citing. The colors used for the overlays are based on Blonde…
Dyads of journals related by citations can agglomerate into specialties through the mechanism of triadic closure. Using the Journal Citation Reports 2011, 2012, and 2013, we analyze triad formation as indicators of integration (specialty growth) and disintegration (restructuring). The strongest integration is found among the large journals that report on studies in different scientific specialties, such as PLoS ONE, Nature Communications, Nature, and Science. This tendency towards large-scale integration has not yet stabilized. Using the Islands algorithm, we also distinguish 51 local maxima o…
Scientific journals are an important choice of publication venue for most authors. Publishing in prestigious journal plays a decisive role for authors in hiring and promotions. In last decade, citation pressure has become intact for all scientific entities more than ever before. Unethical publication practices has started to manipulate widely used performance metric such as "impact factor" for journals and citation based indices for authors. This threatens the integrity of scientific quality and takes away deserved credit of legitimate authors and their authentic publications. In this paper we…
This study examines the correlational relationships between local journal authorship, local and external citation counts, full-text downloads, link-resolver clicks, and four global journal impact factor indices within an all-disciplines journal collection of 12,200 titles and six subject subsets at the University of Illinois at Urbana-Champaign (UIUC) Library. While earlier investigations of the relationships between usage (downloads) and citation metrics have been inconclusive, this study shows strong correlations in the all-disciplines set and most subject subsets. The normalized Eigenfactor…
Single journal studies are reviewed and measures used in the studies are highlighted. The following quantitative measures are used to study 272 articles published in Malaysian Journal of Computer Science, (1) the article productivity of the journal from 1985 to 2007, (2) the observed and expected authorship productivity tested using Lotka's Law of author productivity, identification and listing of core authors; (3) the authorship, co-authorship pattern by authors' country of origin and institutional affiliations; (4) the subject areas of research; (5) the citation analysis of resources referen…
It is shown that the calculation of scientometric indicators of the scientist and also the scientific journal continues to be an actual problem nowadays. It is revealed that the leading scientometric databases have the capabilities of automated metadata collection from the scientific journal website by the use of specialized electronic document management systems, in particular Open Journal Systems. It is established that Open Journal Systems successfully exports metadata about an article from scientific journals to scientometric databases Scopus, Web of Science and Google Scholar. However, th…
Rankings of scholarly journals based on citation data are often met with skepticism by the scientific community. Part of the skepticism is due to disparity between the common perception of journals' prestige and their ranking based on citation counts. A more serious concern is the inappropriate use of journal rankings to evaluate the scientific influence of authors. This paper focuses on analysis of the table of cross-citations among a selection of Statistics journals. Data are collected from the Web of Science database published by Thomson Reuters. Our results suggest that modelling the excha…
A citation-based indicator for interdisciplinarity has been missing hitherto among the set of available journal indicators. In this study, we investigate network indicators (betweenness centrality), journal indicators (Shannon entropy, the Gini coefficient), and more recently proposed Rao-Stirling measures for "interdisciplinarity." The latter index combines the statistics of both citation distributions of journals (vector-based) and distances in citation networks among journals (matrix-based). The effects of various normalizations are specified and measured using the matrix of 8,207 journals …
The aggregated journal-journal citation matrix derived from the Journal Citation Reports 2001 can be decomposed into a unique subject classification by using the graph-analytical algorithm of bi-connected components. This technique was recently incorporated in software tools for social network analysis. The matrix can be assessed in terms of its decomposability using articulation points which indicate overlap between the components. The articulation points of this set did not exhibit a next-order network of 'general science' journals. However, the clusters differ in size and in terms of the in…
The two most used citation impact indicators in the assessment of scientific journals are, nowadays, the impact factor and the h-index. However, both indicators are not field normalized (vary heavily depending on the scientific category) which makes them incomparable between categories. Furthermore, the impact factor is not robust to the presence of articles with a large number of citations, while the h-index depends on the journal size. These limitations are very important when comparing journals of different sizes and categories. An alternative citation impact indicator is the percentage of …
This paper explores a new indicator of journal citation impact, denoted as source normalized impact per paper (SNIP). It measures a journal's contextual citation impact, taking into account characteristics of its properly defined subject field, especially the frequency at which authors cite other papers in their reference lists, the rapidity of maturing of citation impact, and the extent to which a database used for the assessment covers the field's literature. It further develops Eugene Garfield's notions of a field's 'citation potential' defined as the average length of references lists in a…
In December 2016, after several years of development, Elsevier launched a set of transparent, comprehensive, current, and freely-available journal citation metrics called CiteScore metrics. Most of the CiteScore metrics are static, annual indicators calculated from the dynamic Scopus citation index. In the spirit of recent public statements on the responsible use of metrics, we outline the desirable characteristics of journal citation metrics, discuss how we decided on the cited and citing publications years and document types to be used for CiteScore metrics, and detail the precise method of …
This paper investigates the Twitter interaction patterns of journals from the Science Citation Index (SCI) of Master Journal List (MJL). A total of 953,253 tweets extracted from 857 journal accounts, were analyzed in this study. Findings indicate that SCI journals interacted more with each other but much less with journals from other citation indices. The network structure of the communication graph resembled a tight crowd network, with Nature journals playing a major part. Information sources such as news portals and scientific organizations were mentioned more in tweets, than academic journa…
This research analyzes a "who cites whom" matrix in terms of aggregated, journal-journal citations to determine the location of communication studies on the academic spectrum. Using the Journal of Communication as the seed journal, the 2006 data in the Journal Citation Reports are used to map communication studies. The results show that social and experimental psychology journals are the most frequently used sources of information in this field. In addition, several journals devoted to the use and effects of media and advertising are weakly integrated into the larger communication research com…
We analyse the large-scale structure of the journal citation network built from information contained in the Thomson-Reuters Journal Citation Reports. To this end, we take advantage of the network science paraphernalia and explore network properties like density, percolation robustness, average and largest node distances, reciprocity, incoming and outgoing degree distributions, as well as assortative mixing by node degrees. We discover that the journal citation network is a dense, robust, small, and reciprocal world. Furthermore, in and out node degree distributions display long-tails, with fe…
Eigenfactor.org, a journal evaluation tool which uses an iterative algorithm to weight citations (similar to the PageRank algorithm used for Google) has been proposed as a more valid method for calculating the impact of journals. The purpose of this brief communication is to investigate whether the principle of repeated improvement provides different rankings of journals than does a simple unweighted citation count (the method used by ISI).
In this study, the academic status of users of scientific publications in Mendeley is explored in order to analyse the usage pattern of Mendeley users in terms of subject fields, citation and readership impact. The main focus of this study is on studying the filtering capacity of Mendeley readership counts compared to journal citation scores in detecting highly cited WoS publications. Main finding suggests a faster reception of Mendeley readerships as compared to citations across 5 major field of science. The higher correlations of scientific users with citations indicate the similarity betwee…
We demonstrate conclusively that there is no "Open Access Advantage" for papers from the Astrophysical Journal. The two to one citation advantage enjoyed by papers deposited in the arXiv e-print server is due entirely to the nature and timing of the deposited papers. This may have implications for other disciplines.
Ageing of publications, percentage of self-citations, and impact vary from journal to journal within fields of science. The assumption that citation and publication practices are homogenous within specialties and fields of science is invalid. Furthermore, the delineation of fields and among specialties is fuzzy. Institutional units of analysis and persons may move between fields or span different specialties. The match between the citation index and institutional profiles varies among institutional units and nations. The respective matches may heavily affect the representation of the units. No…
Ramanujacharyulu's (1964) Power-Weakness Ratio (PWR) measures impact by recursively multiplying the citation matrix by itself until convergence is reached in both the cited and citing dimensions; the quotient of these values is defined as PWR, whereby "cited" is considered as power and "citing" as weakness. Analytically, PWR is an attractive candidate for measuring journal impact because of its symmetrical handling of the rows and columns in the asymmetrical citation matrix, its recursive algorithm, and its mathematical elegance. In this study, PWR is discussed and critically assessed in relat…
A key feature of questionable journals is a lack of adequate peer review of their articles. Content of thus unknown quality may be utilised by unsuspecting practitioners or incorporated into peer-reviewed research, becoming legitimised. It is therefore necessary to examine the citation patterns of articles in questionable journals to understand the impact and reach of research in questionable journals. Similar research has tended to focus on authors from low- and middle-income countries. As such, this study investigates the profile and impact of research in questionable journals by authors in …
The purpose of citation indexes and metrics is intended to be a measure for scientific innovation and quality for researchers, journals, and institutions. However, those metrics are often prone to abuse and manipulation by excessive and unethical self-citations induced by authors, reviewers, editors, or journals. Identifying whether there are or not legitimate reasons for self-citations is normally determined during the review process, where the participating parts may have intrinsic incentives, rendering the legitimacy of self-citations, after publication, questionable. In this paper, we cond…
Recently, aggregated journal-journal citation networks were made accessible from the perspective of each journal included in the Science Citation Index see (http://www.leydesdorff.net/). The local matrices can be used to inspect the relevant citation environment of a journal using statistical analysis and visualization techniques from social network analysis. The inspection gives an answer to the question what the local impact of this and other journals in the environment is. In this study the citation environment of Angewandte Chemie was analysed. Angewandte Chemie is one of the prime chemist…
We study the statistics of citations made to the indexed Science journals in the Journal Citation Reports during the period 2004-2013 using different measures. We consider different measures which quantify the impact of the journals. To our surprise, we find that the apparently uncorrelated measures, even when defined in an arbitrary manner, show strong correlations. This is checked over all the years considered. Impact factor being one of these measures, the present work raises the question whether it is actually a nearly perfect index as claimed often. In addition we study the distributions …
Microsoft Academic is a free academic search engine and citation index that is similar to Google Scholar but can be automatically queried. Its data is potentially useful for bibliometric analysis if it is possible to search effectively for individual journal articles. This article compares different methods to find journal articles in its index by searching for a combination of title, authors, publication year and journal name and uses the results for the widest published correlation analysis of Microsoft Academic citation counts for journal articles so far. Based on 126,312 articles from 323 …
Two indicators rule them all: Mean and standard deviation used to calculate other journal indicators based on log-normal distribution of citation counts
Citations measure the importance of a publication, and may serve as a proxy for its popularity and quality of its contents. Here we study the distributions of citations to publications from individual academic institutions for a single year. The average number of citations have large variations between different institutions across the world, but the probability distributions of citations for individual institutions can be rescaled to a common form by scaling the citations by the average number of citations for that institution. We find this feature seem to be universal for a broad selection o…
During the last decade, we have witnessed a sustained growth of South Korea's research output in terms of the world share of publications in the Science Citation Index database. However, Korea's citation performance is not yet as competitive as publication performance. In this study, the authors examine the intellectual structure of Korean S&T field based on social network analysis of journal-journal citation data using the ten Korean SCI journals as seed journals. The results reveal that Korean SCI journals function more like publication places, neither research channels nor information sourc…
Methods developed for mapping the journal structures contained in aggregated journal-journal citations in the Science Citation Index are applied to the Chinese Science Citation Database of the Chinese Academy of Sciences. This database covers 991 journals, of which only 37 had originally English titles. Using factor-analytical and graph-analytical techniques we show that this data is dually structured. The main structure is the intellectual organization of the journals in journal groups (as in the international SCI), but the university-based journals provide an institutional layer that orients…
Questions of definition and measurement continue to constrain a consensus on the measurement of interdisciplinarity. Using Rao-Stirling (RS) Diversity produces sometimes anomalous results. We argue that these unexpected outcomes can be related to the use of "dual-concept diversity" which combines "variety" and "balance" in the definitions (ex ante). We propose to modify RS Diversity into a new indicator (DIV) which operationalizes variety, balance, and disparity independently and then combines them ex post. "Balance" can be measured using the Gini coefficient. We apply DIV to the aggregated ci…
The citation impact of Environment and Planning B can be visualized using its citation relations with journals in its environment as the links of a network. The size of the nodes is varied in correspondence to the relative citation impact in this environment. Additionally, one can correct for the effect of within-journal "self"-citations. The network can be partitioned and clustered using algorithms from social network analysis. After transposing the matrix in terms of rows and columns, the citing patterns can be mapped analogously. Citing patterns reflect the activity of the community of auth…
The ability to predict the long-term impact of a scientific article soon after its publication is of great value towards accurate assessment of research performance. In this work we test the hypothesis that good predictions of long-term citation counts can be obtained through a combination of a publication's early citations and the impact factor of the hosting journal. The test is performed on a corpus of 123,128 WoS publications authored by Italian scientists, using linear regression models. The average accuracy of the prediction is good for citation time windows above two years, decreases fo…
Journals were central to Eugene Garfield's research interests. Among other things, journals are considered as units of analysis for bibliographic databases such as the Web of Science (WoS) and Scopus. In addition to disciplinary classifications of journals, journal citation patterns span networks across boundaries to variable extents. Using betweenness centrality (BC) and diversity, we elaborate on the question of how to distinguish and rank journals in terms of interdisciplinarity. Interdisciplinarity, however, is difficult to operationalize in the absence of an operational definition of disc…
