The constantly increasing rate at which scientific papers are published makes it difficult for researchers to identify papers that currently impact the research field of their interest. Hence, approaches to effectively identify papers of high impact have attracted great attention in the past. In this work, we present a method that seeks to rank papers based on their estimated short-term impact, as measured by the number of citations received in the near future. Similar to previous work, our method models a researcher as she explores the paper citation network. The key aspect is that we incorporate an attention-based mechanism, akin to a time-restricted version of preferential attachment, to explicitly capture a researcher's preference to read papers which received a lot of attention recently. A detailed experimental evaluation on four real citation datasets across disciplines, shows that our approach is more effective than previous work in ranking papers based on their short-term impact.
Cross-media Scientific Research Achievements Query based on Ranking Learning
School of Computer Science (National Pilot School of Software Engineering), Beijing University of Posts and Telecommunications（北京邮电大学计算机学院（国家软件学院）；北京智能电信软件与多媒体北京市重点实验室）；Beijing Key Laboratory of Intelligent Telecommunication Software and Multimedia
Recently, the abundance of digital data enabled the implementation of graph based ranking algorithms that provide system level analysis for ranking publications and authors. Here we take advantage of the entire Physical Review publication archive (1893-2006) to construct authors' networks where weighted edges, as measured from opportunely normalized citation counts, define a proxy for the mechanism of scientific credit transfer. On this network we define a ranking method based on a diffusion algorithm that mimics the spreading of scientific credits on the network. We compare the results obtain…
The number, importance, and popularity of rankings measuring innovation performance and the strength and resources of ecosystems that provide its spatial framework are on an increasing trend globally. In addition to influencing the specific decisions taken by economic actors, these rankings significantly impact the development of innovation-related policies at regional, national, and international levels. The importance of startup ecosystems is proven by the growing scientific interest, which is demonstrated by the increasing number of related scientific articles. The concept of the startup ec…
In national research assessment exercises that take the peer review approach, research organizations are evaluated on the basis of a subset of their scientific production. The dimension of the subset varies from nation to nation but is typically set as a proportional function of the number of researchers employed at each research organization. However, scientific fertility varies from discipline to discipline, meaning that the representativeness of such a subset also varies according to discipline. The rankings resulting from the assessments could be quite sensitive to the size of the share of…
We propose a new performance indicator to evaluate the productivity of research institutions by their disseminated scientific papers. The new quality measure includes two principle components: the normalized impact factor of the journal in which paper was published, and the number of citations received per year since it was published. In both components, the scientific impacts are weighted by the contribution of authors from the evaluated institution. As a whole, our new metric, namely, the institutional performance score takes into account both journal based impact and articles specific impac…
Citation recommendation systems for the scientific literature, to help authors find papers that should be cited, have the potential to speed up discoveries and uncover new routes for scientific exploration. We treat this task as a ranking problem, which we tackle with a two-stage approach: candidate generation followed by re-ranking. Within this framework, we adapt to the scientific domain a proven combination based on "bag of words" retrieval followed by re-scoring with a BERT model. We experimentally show the effects of domain adaptation, both in terms of pretraining on in-domain data and ex…
Researchers or students entering a emerging research area are particularly interested in what newly published papers will be most cited and which young researchers will become influential in the future, so that they can catch the most recent advances and find valuable research directions. However, predicting the future importance of scientific articles and authors is challenging due to the dynamic nature of literature networks and evolving research topics. Different from most previous studies aiming to rank the current importance of literatures and authors, we focus on \emph{ranking the future…
To account for strong aging characteristics of citation networks, we modify Google's PageRank algorithm by initially distributing random surfers exponentially with age, in favor of more recent publications. The output of this algorithm, which we call CiteRank, is interpreted as approximate traffic to individual publications in a simple model of how researchers find new information. We develop an analytical understanding of traffic flow in terms of an RPA-like model and optimize parameters of our algorithm to achieve the best performance. The results are compared for two rather different citati…
The launch of Google Scholar Metrics as a tool for assessing scientific journals may be serious competition for Thomson Reuters Journal Citation Reports, and for Scopus powered Scimago Journal Rank. A review of these bibliometric journal evaluation products is performed. We compare their main characteristics from different approaches: coverage, indexing policies, search and visualization, bibliometric indicators, results analysis options, economic cost and differences in their ranking of journals. Despite its shortcomings, Google Scholar Metrics is a helpful tool for authors and editors in ide…
Jadavpur University（贾达普大学）；University of Maryland, College Park（马里兰大学帕克分校）
Whenever human beings interact with each other, they exchange or express opinions, emotions, and sentiments. These opinions can be expressed in text, speech or images. Analysis of these sentiments is one of the popular research areas of present day researchers. Sentiment analysis, also known as opinion mining tries to identify or classify these sentiments or opinions into two broad categories - positive and negative. In recent years, the scientific community has taken a lot of interest in analyzing sentiment in textual data available in various social media platforms. Much work has been done o…
How to Find Fantastic AI Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review
University of Pennsylvania（宾夕法尼亚大学）；Yale University（耶鲁大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；New York University（纽约大学）；Princeton University（普林斯顿大学）
As the volume of electronically available information grows, relevant items become harder to find. This work presents an approach to personalizing search results in scientific publication databases. This work focuses on re-ranking search results from existing search engines like Solr or ElasticSearch. This work also includes the development of Obelix, a new recommendation system used to re-rank search results. The project was proposed and performed at CERN, using the scientific publications available on the CERN Document Server (CDS). This work experiments with re-ranking using offline and onl…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University of Massachusetts Medical School（马萨诸塞大学医学院）
Finding related published articles is an important task in any science, but with the explosion of new work in the biomedical domain it has become especially challenging. Most existing methodologies use text similarity metrics to identify whether two articles are related or not. However biomedical knowledge discovery is hypothesis-driven. The most related articles may not be ones with the highest text similarities. In this study, we first develop an innovative crowd-sourcing approach to build an expert-annotated document-ranking corpus. Using this corpus as the gold standard, we then evaluate t…
We applied a set of standard bibliometric indicators to monitor the scientific state-of-arte of 500 universities worldwide and constructed a ranking on the basis of these indicators (Leiden Ranking 2010). We find a dramatic and hitherto largely underestimated language effect in the bibliometric, citation-based measurement of research performance when comparing the ranking based on all Web of Science (WoS) covered publications and on only English WoS covered publications, particularly for Germany and France.
Although not explicitly declared, most research rankings of countries and institutions are supposed to reveal their contribution to the advancement of knowledge. However, such advances are based on very highly cited publications with very low frequency, which can only very exceptionally be counted with statistical reliability. Percentile indicators enable calculations of the probability or frequency of such rare publications using counts of much more frequent publications; the general rule is that rankings based on the number of top 10% or 1% cited publications (Ptop 10%, Ptop 1%) will also be…
The aim of this report is to present a ranking of Nursing journals covered in Google Scholar Metrics (GSM), a Google product launched in 2012 to assess the impact of scientific journals from citation counts this receive on Google Scholar. Google has chosen to include only those journals that have published at least 100 papers and have at least one citation in a period of five years (2007-2011). Journal rankings are sorted by languages (showing the 100 papers with the greatest impact). This tool allows to sort by subject areas and disciplines, but only in the case of journals in English. In thi…
As the number of scientific journals has multiplied, journal rankings have become increasingly important for scientific decisions. From submissions and subscriptions to grants and hirings, researchers, policy makers, and funding agencies make important decisions with influence from journal rankings such as the ISI journal impact factor. Typically, the rankings are derived from the citation network between a selection of journals and unavoidably depend on this selection. However, little is known about how robust rankings are to the selection of included journals. Here we compare the robustness …
Bornmann, Stefaner, de Moya Anegon, and Mutz (in press) have introduced a web application (www.excellencemapping.net) which is linked to both academic ranking lists published hitherto (e.g. the Academic Ranking of World Universities) as well as spatial visualization approaches. The web application visualizes institutional performance within specific subject areas as ranking lists and on custom tile-based maps. The new, substantially enhanced version of the web application and the multilevel logistic regression on which it is based are described in this paper. Scopus data were used which have b…
What is the value of a scientist and its impact upon the scientific thinking? How can we measure the prestige of a journal or of a conference? The evaluation of the scientific work of a scientist and the estimation of the quality of a journal or conference has long attracted significant interest, due to the benefits from obtaining an unbiased and fair criterion. Although it appears to be simple, defining a quality metric is not an easy task. To overcome the disadvantages of the present metrics used for ranking scientists and journals, J.E. Hirsch proposed a pioneering metric, the now famous h-…
Nowadays impact factor is the significant indicator for journal evaluation. In impact factor calculation is used number of all citations to journal, regardless of the prestige of cited journals, however, scientific units (paper, researcher, journal or scientific organization) cited by journals with high impact factor or researchers with high Hirsch index are more important than objects cited by journals without impact factor or unknown researcher. In this paper was offered weighted impact factor for getting more accurate rankings for journals, which consider not only quantity of citations, but…
Ranking scientific authors is an important but challenging task, mostly due to the dynamic nature of the evolving scientific publications. The basic indicators of an author's productivity and impact are still the number of publications and the citation count (leading to the popular metrics such as h-index, g-index etc.). H-index and its popular variants are mostly effective in ranking highly-cited authors, thus fail to resolve ties while ranking medium-cited and low-cited authors who are majority in number. Therefore, these metrics are inefficient to predict the ability of promising young rese…
Despite the increasing use of citation-based metrics for research evaluation purposes, we do not know yet which metrics best deliver on their promise to gauge the significance of a scientific paper or a patent. We assess 17 network-based metrics by their ability to identify milestone papers and patents in three large citation datasets. We find that traditional information-retrieval evaluation metrics are strongly affected by the interplay between the age distribution of the milestone items and age biases of the evaluated metrics. Outcomes of these metrics are therefore not representative of th…
Quantifying the impact of scientific papers objectively is crucial for research output assessment, which subsequently affects institution and country rankings, research funding allocations, academic recruitment and national/international scientific priorities. While most of the assessment schemes based on publication citations may potentially be manipulated through negative citations, in this study, we explore Conflict of Interest (COI) relationships and discover negative citations and subsequently weaken the associated citation strength. PANDORA (Positive And Negative COI- Distinguished Objec…
It is widely recognized that citation counts for papers from different fields cannot be directly compared because different scientific fields adopt different citation practices. Citation counts are also strongly biased by paper age since older papers had more time to attract citations. Various procedures aim at suppressing these biases and give rise to new normalized indicators, such as the relative citation count. We use a large citation dataset from Microsoft Academic Graph and a new statistical framework based on the Mahalanobis distance to show that the rankings by well known indicators, i…
There exist ample demonstrations that indicators of scholarly impact analogous to the citation-based ISI Impact Factor can be derived from usage data. However, contrary to the ISI IF which is based on citation data generated by the global community of scholarly authors, so far usage can only be practically recorded at a local level leading to community-specific assessments of scholarly impact that are difficult to generalize to the global scholarly community. We define a journal Usage Impact Factor which mimics the definition of the Thomson Scientific's ISI Impact Factor. Usage Impact Factor r…
Predicting the popularity of scientific publications has attracted many attentions from various disciplines. In this paper, we focus on the popularity prediction problem of scientific papers, and propose an age-based diffusion (AD) model to identify which paper will receive more citations in the near future and will be popular. The AD model is a mimic of the attention diffusion process along the citation networks. The experimental study shows that the AD model can achieve better prediction accuracy than other networkbased methods. For some newly published papers that have not accumulated many …
This paper proposes an indicator of journals' scientific prestige, the SJR indicator, for ranking scholarly journals based on citation weighting schemes and eigenvector centrality to be used in complex and heterogeneous citation networks such Scopus. Its computation methodology is described and the results after implementing the indicator over Scopus 2007 dataset are compared to an ad-hoc Journal Impact Factor both generally and inside specific scientific areas. The results showed that SJR indicator and JIF distributions fitted well to a power law distribution and that both metrics were strong…
In benchmarking international research, although publication and citation analyses should not be used to compare different disciplines, scientometrists frequently fail to resist the temptation to present rankings based on total publications and citations. Such measures are affected by significant distortions, due to the uneven fertility across scientific disciplines and the dishomogeneity of scientific specialisation among nations and universities. In this paper, we provide an indication of the extent of the distortions when comparative bibliometric analyses fail to recognise the range of leve…
There is a growing need for ranking universities, departments, research groups, and individual scholars. Usually, the scientific community measures the scientific merits of the researchers by using a variety of indicators that take into account both the productivity of scholars and the impact of their publications. We propose the t-index, the new indicator to measure the scientific merits of the individual researchers. The proposed t-index takes into account the number of citations, number of coauthors on every published paper, and career duration. The t-index makes the possible comparison of …
Rankings of scholarly journals based on citation data are often met with skepticism by the scientific community. Part of the skepticism is due to disparity between the common perception of journals' prestige and their ranking based on citation counts. A more serious concern is the inappropriate use of journal rankings to evaluate the scientific influence of authors. This paper focuses on analysis of the table of cross-citations among a selection of Statistics journals. Data are collected from the Web of Science database published by Thomson Reuters. Our results suggest that modelling the excha…
In this paper, we propose a measure to assess scientific impact that discounts self-citations and does not require any prior knowledge on the their distribution among publications. This index can be applied to both researchers and journals. In particular, we show that it fills the gap of h-index and similar measures that do not take into account the effect of self-citations for authors or journals impact evaluation. The paper provides with two real-world examples: in the former, we evaluate the research impact of the most productive scholars in Computer Science (according to DBLP); in the latt…
Citations between scientific papers and related bibliometric indices, such as the $h$-index for authors and the impact factor for journals, are being increasingly used - often in controversial ways - as quantitative tools for research evaluation. Yet, a fundamental research question remains still open: to which extent do quantitative metrics capture the significance of scientific works? We analyze the network of citations among the $449,935$ papers published by the American Physical Society (APS) journals between 1893 and 2009, and focus on the comparison of metrics built on the citation count…
The impact of individual scientists is commonly quantified using citation-based measures. The most common such measure is the h-index. A scientist's h-index affects hiring, promotion, and funding decisions, and thus shapes the progress of science. Here we report a large-scale study of scientometric measures, analyzing millions of articles and hundreds of millions of citations across four scientific fields and two data platforms. We find that the correlation of the h-index with awards that indicate recognition by the scientific community has substantially declined. These trends are associated w…
The basic indicators of a researcher's productivity and impact are still the number of publications and their citation counts. These metrics are clear, straightforward, and easy to obtain. When a ranking of scholars is needed, for instance in grant, award, or promotion procedures, their use is the fastest and cheapest way of prioritizing some scientists over others. However, due to their nature, there is a danger of oversimplifying scientific achievements. Therefore, many other indicators have been proposed including the usage of the PageRank algorithm known for the ranking of webpages and its…
The impact of scientific publications has traditionally been expressed in terms of citation counts. However, scientific activity has moved online over the past decade. To better capture scientific impact in the digital era, a variety of new impact measures has been proposed on the basis of social network analysis and usage log data. Here we investigate how these new measures relate to each other, and how accurately and completely they express scientific impact. We performed a principal component analysis of the rankings produced by 39 existing and proposed measures of scholarly impact that wer…
Over the last few years, several major scientific fraud cases have shocked the scientific community. The number of retractions each year has also increased tremendously, especially in the biomedical field, and scientific misconduct accounts for approximately more than half of those retractions. It is assumed that co-authors of retracted papers are affected by their colleagues' misconduct, and the aim of this study is to provide empirical evidence of the effect of retractions in biomedical research on co-authors' research careers. Using data from the Web of Science (WOS), we measured the produc…
We performed a citation analysis on the Web of Science publications consisting of more than 63 million articles and 1.45 billion citations on 254 subjects from 1981 to 2020. We proposed the Article's Scientific Prestige (ASP) metric and compared this metric to number of citations (#Cit) and journal grade in measuring the scientific impact of individual articles in the large-scale hierarchical and multi-disciplined citation network. In contrast to #Cit, ASP, that is computed based on the eigenvector centrality, considers both direct and indirect citations, and provides steady-state evaluation c…
The number of citations is a widely used metric to evaluate the scientific credit of papers, scientists and journals. However, it does happen that a paper with fewer citations from prestigious scientists is of higher influence than papers with more citations. In this paper, we argue that from whom the paper is being cited is of higher significance than merely the number of received citations. Accordingly, we propose an interactive model on author-paper bipartite networks as well as an iterative algorithm to get better rankings for scientists and their publications. The main advantage of this m…
A desirable goal of scientific management is to introduce, if it exists, a simple and reliable way to measure the scientific excellence of publicly-funded research institutions and universities to serve as a basis for their ranking and financing. While citation-based indicators and metrics are easily accessible, they are far from being universally accepted as way to automate or inform evaluation processes or to replace evaluations based on peer review. Here we consider absolute measurements of research excellence at an amalgamated, institutional level and specific measures of research excellen…
In this work we have studied the research activity for countries of Europe, Latin America and Africa for all sciences between 1945 and November 2008. All the data are captured from the Web of Science database during this period. The analysis of the experimental data shows that, within a nonextensive thermostatistical formalism, the Tsallis \emph{q}-exponential distribution $N(c)$ satisfactorily describes Institute of Scientific Information citations. The data which are examined in the present survey can be fitted successfully as a first approach by applying a {\it single} curve (namely, $N(c) …
The objective assessment of the prestige of an academic institution is a difficult and hotly debated task. In the last few years, different types of University Rankings have been proposed to quantify the excellence of different research institutions in the world. Albeit met with criticism in some cases, the relevance of university rankings is being increasingly acknowledged: indeed, rankings are having a major impact on the design of research policies, both at the institutional and governmental level. Yet, the debate on what rankings are {\em exactly} measuring is enduring. Here, we address th…
