To what extent can news media help in providing more credible information about science? This is the core challenge for the Science Evidence Indicator (SEI) project, a collaboration between the Danish popular news website videnskab.dk and the authors of this paper. Looking specifically at medical science news, we aim to provide a transparent assessment of the scientific sources behind a story. This entails identifying some of the criteria that scientists use to assess research, and making it accessible and understandable for readers. We address the following research question: How can we communicate the quality of scientific publications in health science to a non-expert audience? Our goal is to make the assessments understandable for the youngest part of the website's target audience: high school students from age 16 and upwards.
Scholars in science and technology studies and bibliometricians are increasingly revealing the performative nature of bibliometric indicators. Far from being neutral technical measures, indicators such as the Impact Factor and the h-index are deeply transforming the social and epistemic structures of contemporary science. At the same time, scholars have highlighted how bibliometric indicators are endowed with social meanings that go beyond their purely technical definitions. These social representations of bibliometric indicators are constructed and negotiated between different groups of actor…
This paper proposes an indicator of journals' scientific prestige, the SJR indicator, for ranking scholarly journals based on citation weighting schemes and eigenvector centrality to be used in complex and heterogeneous citation networks such Scopus. Its computation methodology is described and the results after implementing the indicator over Scopus 2007 dataset are compared to an ad-hoc Journal Impact Factor both generally and inside specific scientific areas. The results showed that SJR indicator and JIF distributions fitted well to a power law distribution and that both metrics were strong…
A new size-independent indicator of scientific journal prestige, the SJR2 indicator, is proposed. This indicator takes into account not only the prestige of the citing scientific journal but also its closeness to the cited journal using the cosine of the angle between the vectors of the two journals' cocitation profiles. To eliminate the size effect, the accumulated prestige is divided by the fraction of the journal's citable documents, thus eliminating the decreasing tendency of this type of indicator and giving meaning to the scores. Its method of computation is described, and the results of…
This paper analyses the potential use of bibliometric data for mapping and applying network analysis to mobility flows. We show case mobility networks at three different levels of aggregation: at the country level, at the city level and at the institutional level. We reflect on the potential uses of bibliometric data to inform research policies with regard to scientific mobility.
Responsible indicators are crucial for research assessment and monitoring. Transparency and accuracy of indicators are required to make research assessment fair and ensure reproducibility. However, sometimes it is difficult to conduct or replicate studies based on indicators due to the lack of transparency in conceptualization and operationalization. In this paper, we review the different variants of the Probabilistic Affinity Index (PAI), considering both the conceptual and empirical underpinnings. We begin with a review of the historical development of the indicator and the different alterna…
Colombian scientific journals are poorly represented in international digital libraries; however, through Google Scholar (GS) it is possible to determine their use by the community. Between the years of 2003 and 2007 a classification of 185 Colombian journals indexed in the Colombian National Bibliographical Index (IBNP) was performed using the information provided by GS, basing categorization on size indicators, indexation and citation. The indicators were analyzed by grouping the journals in two general areas: sciences and social sciences. In each area, the indicators provided by the digital…
In this paper we review the socalled altmetrics or alternative metrics. This concept raises from the development of new indicators based on Web 2.0, for the evaluation of the research and academic activity. The basic assumption is that variables such as mentions in blogs, number of twits or of researchers bookmarking a research paper for instance, may be legitimate indicators for measuring the use and impact of scientific publications. In this sense, these indicators are currently the focus of the bibliometric community and are being discussed and debated. We describe the main platforms and in…
Citation numbers are extensively used for assessing the quality of scientific research. The use of raw citation counts is generally misleading, especially when applied to cross-disciplinary comparisons, since the average number of citations received is strongly dependent on the scientific discipline of reference of the paper. Measuring and eliminating biases in citation patterns is crucial for a fair use of citation numbers. Several numerical indicators have been introduced with this aim, but so far a specific statistical test for estimating the fairness of these numerical indicators has not b…
h-index has become the most popular indicator for quantifying a scientist's scientific impact in various scientific fields. h-index is defined as the largest number of papers with citation number larger than or equal to h and it treats each citation equally. However, different citations usually come from different papers with different influence and quality, and a citation from a highly influential paper is a greater recognition of the target paper than a citation from an ordinary paper. Based on this assumption, we proposed a new indicator named x-index to quantify a scientist's scientific im…
Using bibliometric data artificially generated through a model of citation dynamics calibrated on empirical data, we compare several indicators for the scientific impact of individual researchers. The use of such a controlled setup has the advantage of avoiding the biases present in real databases, and allows us to assess which aspects of the model dynamics and which traits of individual researchers a particular indicator actually reflects. We find that the simple citation average performs well in capturing the intrinsic scientific ability of researchers, whatever the length of their career. O…
In this research, a new indicator of disciplinarity-multidisciplinarity is developed, discussed and applied. EBDI is based on the combination of the frequency distribution of subject categories of journals citing or cited by the analysis unit and the spread and diversity of the citations among subject categories measured with Shannon-Wiener entropy. Its reproducibility, robustness and consistence are discussed. Four of the combinations of its values when applied to the cited and citing dimensions lead to a suggested taxonomy of the role that the studied unit might have in terms of the transfor…
This paper presents and describes the methodological opportunities offered by bibliometric data to produce indicators of scientific mobility. Large bibliographic datasets of disambiguated authors and their affiliations allow for the possibility of tracking the affiliation changes of scientists. Using the Web of Science as data source, we analyze the distribution of types of mobile scientists for a selection of countries. We explore the possibility of creating profiles of international mobility at the country level, and discuss potential interpretations and caveats. Five countries (Canada, The …
COVID-19 is one of the most important topic these days, specifically on search engines and news. While fake news are easily shared, scientific papers are reliable sources where information can be extracted. With about 24,000 scientific publications on COVID-19 and related research on PUBMED, automatic computer-assisted analysis is required. In this paper, we develop two methodologies to get insights on specific sub-topics of interest and latest research sub-topics. They rely on natural language processing and graph-based visualizations. We run these methodologies on two cases: the virus origin…
This paper presents the results and further development of a survey sent to 11,799 Spanish faculty members and researchers from various fields of the social sciences and the humanities, obtaining a total of 45.6% (5,368 responses) usable answers. Respondents were asked (a) to indicate the three most important journals in their field and (b) to rate them on a 0-10 scale according to their quality. The information obtained has been synthesized in two indicators which reflect the perceived quality of journals. Once the values were obtained, the journals were categorized according to each indicato…
The use of quantitative indicators of scientific productivity seems now quite widespread for assessing researchers and research institutions. There is a general perception, however, that these indicators are not necessarily representative of the originality of the research carried out, being primarily indicative of a more or less prolific scientific activity and of the size of the targeted scientific subcommunity. We first discuss some of the drawbacks of the broadly adopted $h$-index and of the fact that it represents, in an average sense, an indicator derivable from the total number of citat…
We address the question how citation-based bibliometric indicators can best be normalized to ensure fair comparisons between publications from different scientific fields and different years. In a systematic large-scale empirical analysis, we compare a traditional normalization approach based on a field classification system with three source normalization approaches. We pay special attention to the selection of the publications included in the analysis. Publications in national scientific journals, popular scientific magazines, and trade magazines are not included. Unlike earlier studies, we …
The two most used citation impact indicators in the assessment of scientific journals are, nowadays, the impact factor and the h-index. However, both indicators are not field normalized (vary heavily depending on the scientific category) which makes them incomparable between categories. Furthermore, the impact factor is not robust to the presence of articles with a large number of citations, while the h-index depends on the journal size. These limitations are very important when comparing journals of different sizes and categories. An alternative citation impact indicator is the percentage of …
Bibliometric mapping and visualization techniques represent one of the main pillars in the field of scientometrics. Traditionally, the main methodologies employed for representing data are Multi-Dimensional Scaling, Principal Component Analysis or Correspondence Analysis. In this paper we aim at presenting a visualization methodology known as Biplot analysis for representing bibliometric and science and technology indicators. A Biplot is a graphical representation of multivariate data, where the elements of a data matrix are represented according to dots and vectors associated with the rows an…
The Italian National Scientific Qualification (ASN) was introduced in 2010 as part of a major reform of the national university system. Under the new regulation, the scientific qualification for a specific role (associate or full professor) and field of study is required to apply to a permanent professor position. The ASN is peculiar since it makes use of bibliometric indicators with associated thresholds as one of the parameters used to assess applicants. Overall, more than 59000 applications were submitted, and the results have been made publicly available for a short period of time, includi…
Scientists are embedded in social and information networks that influence and are influenced by the quality of their scientific work, its impact, and the recognition they receive. Here we quantify the systematic relationship between a scientist's position in the network of scientific collaborations and the citations they receive. As expected, we find that authors closer to others in this network are, on average, more highly cited than those further away from others. We construct a novel indicator, the s-index, that explicitly captures performance linked to network position along two compliment…
In recent years, new indicators known as altmetrics have been introduced to measure the impact of scientific activity. These indicators are obtained through the mentions realised from different social media, existing several aggregators of these data that collect several of them in the same database, being Altmetric.com the most popular. However, in spite of the popularization of these metrics, several limitations in their use have been manifested. For this reason, rhe objective of this work is twofold: (1) to show the possibilities of altimetric techniques applied to the Spanish social scienc…
This article analyses the difference in timing between the online availability of articles and their corresponding print publication and how it affects two bibliometric indicators: Journal Impact Factor (JIF) and Immediacy Index. This research examined 18,526 articles, the complete collection of articles and reviews published by a set of 61 journals on Urology and Nephrology in 2013 and 2014. The findings suggest that Advance Online Publication (AOP) accelerates the citation of articles and affects the JIF and Immediacy Index values. Regarding the JIF values, the comparison between journals wi…
Nowadays impact factor is the significant indicator for journal evaluation. In impact factor calculation is used number of all citations to journal, regardless of the prestige of cited journals, however, scientific units (paper, researcher, journal or scientific organization) cited by journals with high impact factor or researchers with high Hirsch index are more important than objects cited by journals without impact factor or unknown researcher. In this paper was offered weighted impact factor for getting more accurate rankings for journals, which consider not only quantity of citations, but…
Concordia University（康考迪亚大学）；National Research Council Canada（加拿大国家研究委员会）
Early identification of emergent topics is of eminent importance due to their potential impacts on society. There are many methods for detecting emerging terms and topics, all with advantages and drawbacks. However, there is no consensus about the attributes and indicators of emergence. In this study, we evaluate emerging topic detection in the field of artificial intelligence using a new method to evaluate emergence. We also introduce two new attributes of collaboration and technological impact which can help us use both paper and patent information simultaneously. Our results confirm that th…
In this paper an analysis of the presence and possibilities of altmetrics for bibliometric and performance analysis is carried out. Using the web based tool Impact Story, we collected metrics for 20,000 random publications from the Web of Science. We studied both the presence and distribution of altmetrics in the set of publications, across fields, document types and over publication years, as well as the extent to which altmetrics correlate with citation indicators. The main result of the study is that the altmetrics source that provides the most metrics is Mendeley, with metrics on readershi…
This paper examines the scientific performance of South Africa since 1994 (post-apartheid) until 2014 in comparisons with the rest of the world, utilizing relative indicator. It provides a view over current standing of South Africa in the scientific world as well as its temporal evolution after the apartheid. This study focuses on four major aspects of scientific performance, namely quantity, productivity, impact and quality, as the main attributes of scientific perfomance on national level. These are measured by re-based (relative) publication, publication per population or GDP, citations and…
In a critical and provocative paper, Abramo and D'Angelo claim that commonly used scientometric indicators such as the mean normalized citation score (MNCS) are completely inappropriate as indicators of scientific performance. Abramo and D'Angelo argue that scientific performance should be quantified using indicators that take into account the productivity of a research unit. We provide a response to Abramo and D'Angelo, indicating where we believe they raise important issues, but also pointing out where we believe their claims to be too extreme.
In recent years, several Scientometrics and Bibliometrics indicators were proposed to evaluate the scientific impact of individuals, institutions, colleges, universities and research teams. The h-index gives a major breakthrough in the research community to evaluate the scientific impact of an individual. It got a lot of attention due to its simplicity and several other indicators were proposed to extend the properties of h-index as well as to overcome shortcomings of h-index. In this literature review, we have discussed the advantages and limitations of almost all Scientometrics as well as Bi…
The citation potential is a measure of the probability of being cited. Obviously, it is different among fields of science, social science, and humanities because of systematic differences in publication and citation behaviour across disciplines. In the past, the citation potential was studied at journal level considering the average number of references in established groups of journals (for example, the crown indicator is based on the journal subject categories in the Web of Science database). In this paper, some characterizations of the author's scientific research through three different re…
Most of the scientometric indicators use only the total number of citations of an article and produce a single number for scientific assessment of scholars. Although this concept is very simple to compute, it fails to show the scientific productivity and impact of scholars during a time-span or in a year. To overcome this, several time series indicators have been proposed that consider the citations from the entire research career of a scholar. However, these indicators fail to give a comparative assessment of two scholars having same or very similar index value. To overcome this shortcoming, …
Bibliographic metrics are commonly utilized for evaluation purposes within academia, often in conjunction with other metrics. These metrics vary widely across fields and change with the seniority of the scholar; consequently, the only way to interpret these values is by comparison with other academics within the same field who are of similar seniority. Among the field- and time- normalized indicators, rank percentile has grown in popularity, and it is preferred over other types of indicators. In this paper, we propose and justify a novel rank percentile indicator for scholars. Furthermore, we …
Abramo and D'Angelo (in press) doubt the validity of established size-independent indicators measuring citation impact and plead in favor of measuring scientific efficiency (by using the Fractional Scientific Strength indicator). This note is intended to comment on some questionable and a few favorable approaches in the paper by Abramo and D'Angelo (in press).
Identifying important scholarly literature at an early stage is vital to the academic research community and other stakeholders such as technology companies and government bodies. Due to the sheer amount of research published and the growth of ever-changing interdisciplinary areas, researchers need an efficient way to identify important scholarly work. The number of citations a given research publication has accrued has been used for this purpose, but these take time to occur and longer to accumulate. In this article, we use altmetrics to predict the short-term and long-term citations that a s…
Assessing the research performance of multi-disciplinary institutions, where scientists belong to many fields, requires that the evaluators plan how to aggregate the performance measures of the various fields. Two methods of aggregation are possible. These are based on: a) the performance of the individual scientists or b) the performance of the scientific fields present in the institution. The appropriate choice depends on the evaluation context and the objectives for the particular measure. The two methods bring about differences in both the performance scores and rankings. We quantify these…
A bibliometric approach is explored to tracking international scientific migration, based on an analysis of the affiliation countries of authors publishing in peer reviewed journals indexed in Scopus. The paper introduces a model that relates base concepts in the study of migration to bibliometric constructs, and discusses the potentialities and limitations of a bibliometric approach both with respect to data accuracy and interpretation. Synchronous and asynchronous analyses are presented for 10 rapidly growing countries and 7 scientifically established countries. Rough error rates of the prop…
The h-index is a popular bibliometric indicator for assessing individual scientists. We criticize the h-index from a theoretical point of view. We argue that for the purpose of measuring the overall scientific impact of a scientist (or some other unit of analysis) the h-index behaves in a counterintuitive way. In certain cases, the mechanism used by the h-index to aggregate publication and citation statistics into a single number leads to inconsistencies in the way in which scientists are ranked. Our conclusion is that the h-index cannot be considered an appropriate indicator of a scientist's …
The Italian National Scientific Qualification (ASN) was introduced as a prerequisite for applying for tenured associate or full professor positions at state-recognized universities. The ASN is meant to attest that an individual has reached a suitable level of scientific maturity to apply for professorship positions. A five member panel, appointed for each scientific discipline, is in charge of evaluating applicants by means of quantitative indicators of impact and productivity, and through an assessment of their research profile. Many concerns were raised on the appropriateness of the evaluati…
How to quantify the impact of a researcher's or an institution's body of work is a matter of increasing importance to scientists, funding agencies, and hiring committees. The use of bibliometric indicators, such as the h-index or the Journal Impact Factor, have become widespread despite their known limitations. We argue that most existing bibliometric indicators are inconsistent, biased, and, worst of all, susceptible to manipulation. Here, we pursue a principled approach to the development of an indicator to quantify the scientific impact of both individual researchers and research institutio…
We study the distributions of citations received by a single publication within several disciplines, spanning broad areas of science. We show that the probability that an article is cited $c$ times has large variations between different disciplines, but all distributions are rescaled on a universal curve when the relative indicator $c_f=c/c_0$ is considered, where $c_0$ is the average number of citations per article for the discipline. In addition we show that the same universal behavior occurs when citation distributions of articles published in the same field, but in different years, are com…
In Hungary, the highest and most prestigious scientific qualification is considered to be the Doctor of Science (DSc) title being awarded by the Hungarian Academy of Sciences. The academic performance indicators of the DSc title are of high importance in the evaluation of individuals' research performance not only when a researcher applies for obtaining a DSc title, but also during promotions and appointments at universities, and in the case of the evaluation of applications for scientific titles and degrees, and the assessment of applications for funding. In the Section of Earth Sciences enco…
In the last couple of years, the role of Open Access (OA) publishing has become central in science management and research policy. In the UK and the Netherlands, national OA mandates require the scientific community to seriously consider publishing research outputs in OA forms. At the same time, other elements of Open Science are becoming also part of the debate, thus including not only publishing research outputs but also other related aspects of the chain of scientific knowledge production such as open peer review and open data. From a research management point of view, it is important to ke…
An extensive analysis of the presence of different altmetric indicators provided by Altmetric.com across scientific fields is presented, particularly focusing on their relationship with citations. Our results confirm that the presence and density of social media altmetric counts are still very low and not very frequent among scientific publications, with 15%-24% of the publications presenting some altmetric activity and concentrating in the most recent publications, although their presence is increasing over time. Publications from the social sciences, humanities and the medical and life scien…
The enlarged coverage of the international publication and citation databases Web of Science and Scopus towards local media in social sciences was a welcome response to an increased usage of these databases in evaluation and funding systems. The mostly international journals available earlier were the basis for the development of current standard bibliometric indicators. The same indicators may no longer measure exactly the same concepts when applied to newly introduced or extended media categories, with possibly different characteristics than those of international journals. This paper invest…
We propose a new performance indicator to evaluate the productivity of research institutions by their disseminated scientific papers. The new quality measure includes two principle components: the normalized impact factor of the journal in which paper was published, and the number of citations received per year since it was published. In both components, the scientific impacts are weighted by the contribution of authors from the evaluated institution. As a whole, our new metric, namely, the institutional performance score takes into account both journal based impact and articles specific impac…
Following in the footsteps of the model of scientific communication, which has recently gone through a metamorphosis (from the Gutenberg galaxy to the Web galaxy), a change in the model and methods of scientific evaluation is also taking place. A set of new scientific tools are now providing a variety of indicators which measure all actions and interactions among scientists in the digital space, making new aspects of scientific communication emerge. In this work we present a method for capturing the structure of an entire scientific community (the Bibliometrics, Scientometrics, Informetrics, W…
Recently, two new indicators (Equalized Mean-based Normalized Proportion Cited, EMNPC; Mean-based Normalized Proportion Cited, MNPC) were proposed which are intended for sparse scientometrics data. The indicators compare the proportion of mentioned papers (e.g. on Facebook) of a unit (e.g., a researcher or institution) with the proportion of mentioned papers in the corresponding fields and publication years (the expected values). In this study, we propose a third indicator (Mantel-Haenszel quotient, MHq) belonging to the same indicator family. The MHq is based on the MH analysis - an establish…
Accurately linking news articles to scientific research works is a critical component in a number of applications, such as measuring the social impact of a research work and detecting inaccuracies or distortions in science news. Although the lack of links between news and literature has been a challenge in these applications, it is a relatively unexplored research problem. In this paper we designed and evaluated a new approach that consists of (1) augmenting latest named-entity recognition techniques to extract various metadata, and (2) designing a new elastic search engine that can facilitate…
Background: Conferences bring scientists together and provide one of the most timely means for disseminating new ideas and cutting-edge works.The importance of conferences in scientific areas is testified by quantitative indicators. In Computer Science, for instance, almost two out of three papers published on Scopus are conference papers. Objective/Purpose: The main goal of this paper is to investigate a novel research question: is there any correlation between the impact of a scientific conference and the venue where it took place? Approach: In order to measure the impact of conferences we c…
There has been a proliferation of descriptive for COVID-19 papers using altmetrics. The main objective of this study is to analyse whether the altmetric mentions of COVID-19 medical studies are associated with the type of study and its level of evidence. Data were collected from PubMed and Altmetric.com databases. A total of 16,672 study types (e.g., Case reports or Clinical trials) published in the year 2021 and with at least one altmetric mention were retrieved. The altmetric indicators considered were Altmetric Attention Score (AAS), News mentions, Twitter mentions, and Mendeley readers. On…
The article "Caveats for the journal and field normalizations in the CWTS (`Leiden') evaluations of research performance", published by Tobias Opthof and Loet Leydesdorff (arXiv:1002.2769) deals with a subject as important as the application of so called field normalized indicators of citation impact in the assessment of research performance of individual researchers and research groups. Field normalization aims to account for differences in citation practices across scientific-scholarly subject fields. As the primary author of the papers presenting the "Leiden" indicators and of many reports …
