Do open citations give insights on the qualitative peer-review evaluation in research assessments? An analysis of the Italian National Scientific Qualification
Federica Bologna, Angelo Di Iorio, Silvio Peroni, Francesco Poggi
In the past, several works have investigated ways for combining quantitative and qualitative methods in research assessment exercises. Indeed, the Italian National Scientific Qualification (NSQ), i.e. the national assessment exercise which aims at deciding whether a scholar can apply to professorial academic positions as Associate Professor and Full Professor, adopts a quantitative and qualitative evaluation process: it makes use of bibliometrics followed by a peer-review process of candidates' CVs. The NSQ divides academic disciplines into two categories, i.e. citation-based disciplines (CDs) and non-citation-based disciplines (NDs), a division that affects the metrics used for assessing the candidates of that discipline in the first part of the process, which is based on bibliometrics. In this work, we aim at exploring whether citation-based metrics, calculated only considering open bibliographic and citation data, can support the human peer-review of NDs and yield insights on how it is conducted. To understand if and what citation-based (and, possibly, other) metrics provide relevant information, we created a series of machine learning models to replicate the decisions of the NSQ committees. As one of the main outcomes of our study, we noticed that the strength of the citational relationship between the candidate and the commission in charge of assessing his/her CV seems to play a role in the peer-review phase of the NSQ of NDs.
Assessing evaluation procedures for individual researchers: the case of the Italian National Scientific Qualification
意大利国家科学资格（英语：Italian National Scientific Qualification）是申请国立大学终身副教授或教授职位的先决条件。该证书旨在证明一个人已经达到了合适的科学成熟度水平，可以申请教授职位。为每个科学学科指定的五人小组负责通过影响和生产力的量化指标以及通过评估其研究概况来评估申请人。对评价标准的适当性，特别是对使用文献计量学评价个别研究人员的问题提出了许多关切。其他关切涉及到最后评价报告被认为质量不佳。在本文中，我们评估的应用方法的适当性，并提供给申请人的反馈质量的条件下的benefit。我们认为，在使用文献计量学指标评价个别研究人员的最佳做法是不完全符合的，此外，最终报告的质量有很大的差异，在各小组，建议采取措施，以防止草率的做法，在未来的研究。
Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011-2014)
The false myth of the rise in self-citations, and the impressively positive effect of bibliometric evaluations on the increase of the impact of Italian research
Knowledge spillovers: does the geographic proximity effect decay over time? A discipline-level analysis, accounting for cognitive proximity, with and without self-citations
On the relation between the degree of internationalization of cited and citing publications: A field level analysis, including and excluding self-citations
