The present review of bibliometric counting methods investigates 1) the number of unique counting methods in the bibliometric research literature, 2) to what extent the counting methods can be categorized according to selected characteristics of the counting methods, 3) methods and elements to assess the internal validity of the counting methods, and 4) to what extent and with which characteristics the counting methods are used in research evaluations.
The review identifies 32 counting methods introduced during the period 1981 - 2018. Two frameworks categorize these counting methods. Framework 1 describes selected mathematical properties of counting methods, and Framework 2 describes arguments for choosing a counting method. Twenty of the 32 counting methods are rank-dependent, fractionalized, and introduced to measure contribution, participation, etc. of an object of study. Next, three criteria for internal validity are used to identify five methods that test the adequacy of counting methods, two elements that test sensitivity, and three elements that test homogeneity of the counting methods. These methods and elements may be used to assess the internal validity of counting methods. Finally, a literature search finds research evaluations that use the counting methods. Only three of the 32 counting methods are used by four research evaluations or more. Of these three counting methods, two are used with the same characteristics as defined in the studies that introduced the counting methods.
The review provides practitioners in research evaluation and researchers in bibliometrics with a detailed foundation for working with counting methods. At the same time, many of the findings in the review provide bases for future investigations of counting methods.
In their study entitled "Constructing bibliometric networks: A comparison between full and fractional counting," Perianes-Rodriguez, Waltman, & van Eck (2016; henceforth abbreviated as PWvE) provide arguments for the use of fractional counting at the network level as different from the level of publications. Whereas fractional counting in the latter case divides the credit among co-authors (countries, institutions, etc.), fractional counting at the network level can normalize the relative weights of links and thereby clarify the structures in the network. PWvE, however, propose a counting sche…
The discussion about counting methods in bibliometrics is often reduced to the choice between full and fractional counting. However, several studies document that this distinction is too simple. The aim of the present study is to give an overview of counting methods in the bibliometric literature and to provide insight into their properties and use. A mix of methods is used. In the preliminary results, a literature review covering 1970-2018 identified 29 original counting methods. Seventeen were introduced in the period 2010-2018. Twenty-one of the 29 counting methods are rank-dependent and fr…
The analysis of bibliometric networks, such as co-authorship, bibliographic coupling, and co-citation networks, has received a considerable amount of attention. Much less attention has been paid to the construction of these networks. We point out that different approaches can be taken to construct a bibliometric network. Normally the full counting approach is used, but we propose an alternative fractional counting approach. The basic idea of the fractional counting approach is that each action, such as co-authoring or citing a publication, should have equal weight, regardless of for instance t…
Development of bibliometric techniques has reached such a level as to suggest their integration or total substitution for classic peer review in the national research assessment exercises, as far as the hard sciences are concerned. In this work we compare rankings lists of universities captured by the first Italian evaluation exercise, through peer review, with the results of bibliometric simulations. The comparison shows the great differences between peer review and bibliometric rankings for excellence and productivity.
An increasing demand for bibliometric assessment of individuals has led to a growth of new bibliometric indicators as well as new variants or combinations of established ones. The aim of this review is to contribute with objective facts about the usefulness of bibliometric indicators of the effects of publication activity at the individual level. This paper reviews 108 indicators that can potentially be used to measure performance on the individual author level, and examines the complexity of their calculations in relation to what they are supposed to reflect and ease of end-user application.
Using co-authored publications between China and Korea in Web of Science (WoS) during the one-year period of 2014, we evaluate the government stimulation program for collaboration between China and Korea. In particular, we apply dual approaches, full integer vs. fractional counting, to collaborative publications in order to better examine both the patterns and contents of Sino-Korean collaboration networks in terms of individual countries and institutions. We first conduct a semi-automatic network analysis of Sino-Korean publications based on the full-integer counting method, and then compare …
National research assessment exercises are becoming regular events in ever more countries. The present work contrasts the peer-review and bibliometrics approaches in the conduct of these exercises. The comparison is conducted in terms of the essential parameters of any measurement system: accuracy, robustness, validity, functionality, time and costs. Empirical evidence shows that for the natural and formal sciences, the bibliometric methodology is by far preferable to peer-review. Setting up national databases of publications by individual authors, derived from Web of Science or Scopus databas…
Bibliometric studies often rely on field-normalized citation impact indicators in order to make comparisons between scientific fields. We discuss the connection between field normalization and the choice of a counting method for handling publications with multiple co-authors. Our focus is on the choice between full counting and fractional counting. Based on an extensive theoretical and empirical analysis, we argue that properly field-normalized results cannot be obtained when full counting is used. Fractional counting does provide results that are properly field normalized. We therefore recomm…
Systematic evaluations of publicly funded research typically employ a combination of bibliometrics and peer review, but it is not known whether the bibliometric component introduces biases. This article compares three alternative mechanisms for scoring 73,612 UK Research Excellence Framework (REF) journal articles from all 34 field-based Units of Assessment (UoAs) 2014-17: peer review, field normalised citations, and journal average field normalised citation impact. All three were standardised into a four-point scale. The results suggest that in almost all academic fields, bibliometric scoring…
Following in the footsteps of the model of scientific communication, which has recently gone through a metamorphosis (from the Gutenberg galaxy to the Web galaxy), a change in the model and methods of scientific evaluation is also taking place. A set of new scientific tools are now providing a variety of indicators which measure all actions and interactions among scientists in the digital space, making new aspects of scientific communication emerge. In this work we present a method for capturing the structure of an entire scientific community (the Bibliometrics, Scientometrics, Informetrics, W…
The peer review system has been traditionally challenged due to its many limitations especially for allocating funding. Bibliometric indicators may well present themselves as a complement. Objective: We analyze the relationship between peers' ratings and bibliometric indicators for Spanish researchers in the 2007 National R&D Plan for 23 research fields. We analyze peers' ratings for 2333 applications. We also gathered principal investigators' research output and impact and studied the differences between accepted and rejected applications. We used the Web of Science database and focused on th…
In this work, we try to answer the question of which method, peer review vs bibliometrics, better predicts the future overall scholarly impact of scientific publications. We measure the agreement between peer review evaluations of Web of Science indexed publications submitted to the first Italian research assessment exercise and long-term citations of the same publications. We do the same for an early citation-based indicator. We find that the latter shows stronger predictive power, i.e., it more reliably predicts late citations in all the disciplinary areas examined, and for any citation time…
Many contemporary research funding instruments and research policies aim for excellence at the level of individual scientists, teams or research programmes. Good bibliometric approximations of related specialties could be useful for instance to help assign reviewers to applications. This paper reports findings on the usability of reviewer suggestions derived from a recently developed specialty approximation method combining key sources, title words, authors and references (Rons, 2018). Reviewer suggestions for applications for Senior Research Fellowships were made available to the evaluation c…
On the Alignment of Post-Publication Reviews & Bibliometric and Altmetric Impact -- A Case Study on Expert Statements from the Science Media Center Germany
Bibliometrics offers a particular representation of science. Through bibliometric methods a bibliometrician will always highlight particular elements of publications, and through these elements operationalize particular representations of science, while obscuring other possible representations from view. Understanding bibliometrics as representation implies that a bibliometric analysis is always performative: a bibliometric analysis brings a particular representation of science into being that potentially influences the science system itself. In this review we analyze the ways the humanities h…
The use of bibliometric indicators would simplify research assessments. The 2014 Research Excellence Framework (REF) is a peer review assessment of UK universities, whose results can be taken as benchmarks for bibliometric indicators. In this study we use the REF results to investigate whether the ep index and a top percentile of most cited papers could substitute for peer review. The probability that a random university's paper reaches a certain top percentile in the global distribution of papers is a power of the ep index, which can be calculated from the citation-based distribution of unive…
Objective. The purpose of this work is to analyse the knowledge structure and trends in scientific research in the Online Information Reviews journal by bibliometric analysis of key words and social network analysis of co-words. Methods. Key words included in a set of 758 papers included in the Web of Science database from 2000 to 2014 were analysed. We conducted a subject analysis considering the key words assigned to papers. A social network analysis was also conducted to identify the number of co-occurrences between key words (co-words). The Pajek software was used to create and graphically…
The application of digital technologies in agriculture can improve traditional practices to adapt to climate change, reduce Greenhouse Gases (GHG) emissions, and promote a sustainable intensification for food security. Some authors argued that we are experiencing a Digital Agricultural Revolution (DAR) that will boost sustainable farming. This study aims to find evidence of the ongoing DAR process and clarify its roots, what it means, and where it is heading. We investigated the scientific literature with bibliometric analysis tools to produce an objective and reproducible literature review. W…
In the past decades, many countries have started to fund academic institutions based on the evaluation of their scientific performance. In this context, post-publication peer review is often used to assess scientific performance. Bibliometric indicators have been suggested as an alternative to peer review. A recurrent question in this context is whether peer review and metrics tend to yield similar outcomes. In this paper, we study the agreement between bibliometric indicators and peer review based on a sample of publications submitted for evaluation to the national Italian research assessment…
The use of outcome control modes of research evaluation exercises is ever more frequent. They are conceived as tools to stimulate increased levels of research productivity, and to guide choices in allocating components of government research budgets for publicly funded institutions. There are several contributions in the literature that compare the different methodological approaches that policy makers could adopt for these exercises, however the comparisons are limited to only a few disciplines. This work, examining the case of the whole of the "hard sciences" of the Italian academic system, …
An important issue in bibliometrics is the weighing of co-authorship in the production of scientific collaborations, which are becoming the standard modality of research activity in many disciplines. The problem is especially relevant in the field of high-energy physics, where collaborations reach 3000 authors, but it can no longer be ignored also in other domains, like medicine or biology. We present theoretical and numerical arguments in favour of weighing the individual contributions as $1/N_{\rm aut}^α$ where $N_{\rm aut}$ is the number of co-authors. When counting citations we suggest the…
Weak measurement has enabled fundamental studies in both experiment and theory of quantum measurement. Numerous researches have indicated that weak measurements have a wide range of application and scientific research value. In our work, we used bibliometric methods to evaluate the global scientific output of research on Weak measurement and explore the current status and trends in this field from 2000 to 2020. The R bibliometric package was used for quantitative and qualitative analyses of publication outputs and author contributions. In total, 636 related articles and reviews were included i…
In December 2003, seventeen years after the first UK research assessment exercise, Italy started up its first-ever national research evaluation, with the aim to evaluate, using the peer review method, the excellence of the national research production. The evaluation involved 20 disciplinary areas, 102 research structures, 18,500 research products and 6,661 peer reviewers (1,465 from abroad); it had a direct cost of 3.55 millions Euros and a time length spanning over 18 months. The introduction of ratings based on ex post quality of output and not on ex ante respect for parameters and complian…
Enriching meta-analyses through scoping review, bibliometrics, and alternative impact metrics: Visualizing study characteristics, hidden risk of bias, societal influence, and research translation
Opthof and Leydesdorff [arXiv:1102.2569] reanalyze data reported by Van Raan [arXiv:physics/0511206] and conclude that there is no significant correlation between on the one hand average citation scores measured using the CPP/FCSm indicator and on the other hand the quality judgment of peers. We point out that Opthof and Leydesdorff draw their conclusions based on a very limited amount of data. We also criticize the statistical methodology used by Opthof and Leydesdorff. Using a larger amount of data and a more appropriate statistical methodology, we do find a significant correlation between t…
This paper presents an analysis of the Overton policy document database, describing the makeup of materials indexed and the nature in which they cite academic literature. We report on various aspects of the data, including growth, geographic spread, language representation, the range of policy source types included, and the availability of citation links in documents. Longitudinal analysis over established journal category schemes is used to reveal the scale and disciplinary focus of citations and determine the feasibility of developing field-normalized citation indicators. We examine how well…
This review summarizes papers which analyze impact of self-citation on research evaluation. We introduce a generalized definition of self-citation and its variants: author, institutional, country, journal, discipline, publisher self-citation. Formulae of the basic self-citation measures are given, namely self-citing and self-cited rates. World literature on author, institutional, country and journal self-citation is studied in more detail. Current views on the role and impact of self-citation are compiled and analyzed. It is found that there is a general consensus on some points: (a) pathologi…
Information exposing, in conjunction with technological innovations and the emergence of social media, altered the traditional roles of academic libraries and enabled librarians to become necessary partners in research. The role of academic librarians in conducting systematic reviews is well recognised, however, their role in conducting scoping reviews is not yet well established. Nevertheless, we propose that, in more and more frequent situations when it is not feasible to read and analyse all relevant literature to be scoped manually, librarians employ bibliometric analysis and mapping to vi…
National exercises for the evaluation of research activity by universities are becoming regular practice in ever more countries. These exercises have mainly been conducted through the application of peer-review methods. Bibliometrics has not been able to offer a valid large-scale alternative because of almost overwhelming difficulties in identifying the true author of each publication. We will address this problem by presenting a heuristic approach to author name disambiguation in bibliometric datasets for large-scale research assessments. The application proposed concerns the Italian universi…
Humans can learn from very few samples, demonstrating an outstanding generalization ability that learning algorithms are still far from reaching. Currently, the most successful models demand enormous amounts of well-labeled data, which are expensive and difficult to obtain, becoming one of the biggest obstacles to the use of machine learning in practice. This scenario shows the massive potential for Transfer Learning, which aims to harness previously acquired knowledge to the learning of new tasks more effectively and efficiently. In this systematic review, we apply a quantitative method to se…
There has been ample demonstration that bibliometrics is superior to peer-review for national research assessment exercises in the hard sciences. In this paper we examine the Italian case, taking the 2001-2003 university performance rankings list based on bibliometrics as benchmark. We compare the accuracy of the first national evaluation exercise, conducted entirely by peer-review, to other rankings lists prepared at zero cost, based on indicators indirectly linked to performance or available on the Internet. The results show that, for the hard sciences, the costs of conducting the Italian ev…
Academic leadership is essential for research innovation and impact. Until now, there has been no dedicated measure of leadership by bibliometrics. Popular bibliometric indices are mainly based on academic output, such as the journal impact factor and the number of citations. Here we develop an academic leadership index based on readily available bibliometric data that is sensitive to not only academic output but also research efficiency. Our leadership index was tested in two studies on peer-reviewed journal papers by extramurally-funded principal investigators in the field of life sciences f…
Google Scholar has been well received by the research community. Its promises of free, universal and easy access to scientific literature as well as the perception that it covers better than other traditional multidisciplinary databases the areas of the Social Sciences and the Humanities have contributed to the quick expansion of Google Scholar Citations and Google Scholar Metrics: two new bibliometric products that offer citation data at the individual level and at journal level. In this paper we show the results of a experiment undertaken to analyze Google Scholar's capacity to detect citati…
During the Italian research assessment exercise, the national agency ANVUR performed an experiment to assess agreement between grades attributed to journal articles by informed peer review (IR) and by bibliometrics. A sample of articles was evaluated by using both methods and agreement was analyzed by weighted Cohen's kappas. ANVUR presented results as indicating an overall 'good' or 'more than adequate' agreement. This paper re-examines the experiment results according to the available statistical guidelines for interpreting kappa values, by showing that the degree of agreement, always in the…
A publication trend in Physics Education by employing bibliometric analysis leads the researchers to describe current scientific movement. This paper tries to answer "What do Physics education scientists concentrate in their publications?" by analyzing the productivity and development of publications on the subject category of Physics Education in the period 1980--2013. The Web of Science databases in the research areas of "EDUCATION - EDUCATIONAL RESEARCH" was used to extract the publication trends. The study involves 1360 publications, including 840 articles, 503 proceedings paper, 22 review…
Blockchain is a shared, immutable ledger that has attracted the attention of researchers and practitioners across innumerable sectors, with its implications for modernizing payment systems having the possibility of inciting a digital revolution. In the scope of this study, 1,511 publications were obtained from Scopus to conduct a systematic review of the research space through bibliometric and network analyses. The main aim of this study was to determine key authors, significant studies, and collaboration patterns, to reveal the distributions and impacts of publications in the blockchain-based…
Research funding and reputation in the UK have, for over two decades, been increasingly dependent on a regular peer-review of all UK departments. This is to move to a system more based on bibliometrics. Assessment exercises of this kind influence the behavior of institutions, departments and individuals, and therefore bibliometrics will have effects beyond simple measurement. [arXiv:0910.3529]
Future Generation Computer Systems (FGCS), published by Elsevier, is a prestigious peer reviewed journal that started in 1984. As on date of writing this article, the journal is in its 137th volume. Owing to its publication quality and continued academic standards, it has been indexed by major academic databases such as Web of Science, Scopus, DOAJ etc. It is now ranked among the top journals in the area of Computer Science (General). Motivated by the long history, academic reputation and prestige of the journal, the present study attempts to do a bibliometric and text-based analysis of public…
Unleashing the Power of AI. A Systematic Review of Cutting-Edge Techniques in AI-Enhanced Scientometrics, Webometrics, and Bibliometrics
Tarbiat Modares University（塔比阿特莫达雷斯大学）；Alzahra University（阿尔扎赫拉大学）；Université de Montreal（蒙特利尔大学）；University of Leicester（莱斯特大学）；Bucharest University of Economic Studies（布加勒斯特经济研究大学）
As the number of empirical studies increases unprecedentedly in line with expansions in higher education, theoretical developments in population studies suffer due to a discoverability crisis of related work. The systematic use of previous research is common in the medical sciences through various types of structured reviews. However, these are less common in the social sciences, despite their potential, especially in cross-disciplinary fields such as migration. We use Google Scholar to examine the niche of housing research within migration studies through a broad range of documents. The contr…
This study uses the article content and metadata of four important computer networking periodicals-IEEE Communications Surveys and Tutorials (COMST), IEEE/ACM Transactions on Networking (TON), ACM Special Interest Group on Data Communications (SIGCOMM), and IEEE International Conference on Computer Communications (INFOCOM)-obtained using ACM, IEEE Xplore, Scopus and CrossRef, for an 18-year period (2000-2017) to address important bibliometrics questions. All of the venues are prestigious, yet they publish quite different research. The first two of these periodicals (COMST and TON) are highly r…
Objective. To analyse quantitatively the articles published during 2008_2017 about the academic social networking site ResearchGate. Methods. A scoping bibliometric review of documents retrieved using Google Scholar was conducted, limited to publications that contained the word "ResearchGate" in their title and were published from 2008 to 2017. Results. The search yielded 159 documents, once a preliminary list of 386 documents retrieved from Google Scholar was filtered, which eliminated about 60% of the results that were bibliographic citations and not documents. Papers in journals were the mo…
This paper presents the results of a systematic review of the literature on the impact of social media in learning and teaching through bibliometric based Citation analysis. The objective of the review was to map the evolution of the current literature and identify the leading sources of knowledge in terms of the most influential journals, authors, and articles. From a total of 50 top most relevant articles selected from the Scopus database, a detailed citation analysis was conducted. The study explored the overall theoretical foundation of social media research involving in learning and study…
In this chapter, we focus on the specialists who have helped to improve the conditions for book assessments in research evaluation exercises, with empirically based data and insights supporting their greater integration. Our review highlights the research carried out by four types of expert communities, referred to as the monitors, the subject classifiers, the indexers and the indicator constructionists. Many challenges lie ahead for scholars affiliated with these communities, particularly the latter three. By acknowledging their unique, yet interrelated roles, we show where the greatest poten…
Bibliometric analysis has firmly conquered its place as an instrument for evaluation and international comparison of performance levels. Consequently, differences in coverage by standard bibliometric databases installed a dichotomy between on the one hand the well covered 'exact' sciences, and on the other hand most of the social sciences and humanities with a more limited coverage (Nederhof, 2006). Also the latter domains need to be able to soundly demonstrate their level of performance and claim or legitimate funding accordingly. An important part of the output volume in social sciences appe…
Citation impact indicators nowadays play an important role in research evaluation, and consequently these indicators have received a lot of attention in the bibliometric and scientometric literature. This paper provides an in-depth review of the literature on citation impact indicators. First, an overview is given of the literature on bibliographic databases that can be used to calculate citation impact indicators (Web of Science, Scopus, and Google Scholar). Next, selected topics in the literature on citation impact indicators are reviewed in detail. The first topic is the selection of public…
