National research evaluation exercises provide a comparative measure of research performance of the nation's institutions, and as such represent a tool for stimulating research productivity, particularly if the results are used to inform selective funding by government. While a school of thought welcomes frequent changes in evaluation criteria in order to prevent the subjects evaluated from adopting opportunistic behaviors, it is evident that the "rules of the game" should above all be functional towards policy objectives, and therefore be known with adequate forewarning prior to the evaluation period. Otherwise, the risk is that policy-makers will find themselves faced by a dilemma: should they reward universities that responded best to the criteria in effect at the outset of the observation period or those that result as best according to rules that emerged during or after the observation period? This study verifies if and to what extent some universities are penalized instead of rewarded for good behavior, in pursuit of the objectives of the "known" rules of the game, by comparing the research performances of Italian universities for the period of the nation's next evaluation exercise (2004-2008): first as measured according to criteria available at the outset of the period and next according to those announced at the end of the period.
Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011-2014)
Evaluating research: from informed peer review to bibliometrics
国家研究评估活动正在越来越多的国家成为定期活动。目前的工作对比同行评议和文献计量学的方法在这些练习的进行。比较是根据任何测量系统的基本参数进行的：准确性，鲁棒性，有效性，功能性，时间和成本。经验证据表明，就自然科学和正规科学而言，文献计量方法远比同行评议更可取。建立来自Web of Science或Scopus数据库的个人作者出版物的国家数据库，将使国家研究评估更好、更便宜和更频繁。
The VQR, Italy's second national research assessment: Methodological failures and ranking distortions
2013年7月完成的2004-2010年VQR是意大利的第二次国家研究评估活动。VQR性能评估遵循了其他国家的模式，即基于选定的产品子集。在这项工作中，我们确定了这项工作的方法上的弱点，并衡量他们在大学绩效排名中造成的扭曲。首先，我们创建一个场景，在这个场景中，我们假设有效的选择的产品提交的大学，并从这个模拟一组排名应用精确的VQR评级标准。接下来，我们比较这些“VQR排名”与那些将来自更适当的文献计量学的应用。最后，我们将比较扩展到基于该时期整个科学生产的大学排名，如在Web of Science中索引的那样。
The dynamically changing publication universe as a reference point in national impact evaluation: A counterfactual case study on the Chinese publication growth
What Large-Scale Publication and Citation Data Tell Us About International Research Collaboration in Europe: Changing National Patterns in Global Contexts
Do open citations give insights on the qualitative peer-review evaluation in research assessments? An analysis of the Italian National Scientific Qualification
Errors and secret data in the Italian research assessment exercise. A comment to a reply
意大利采用了一种基于业绩的大学供资制度，其中心是由一个政府机构（ANVUR）实施的国家研究评估活动的结果。ANVUR使用“双重评估系统”评估论文，即通过知情同行评审或文献计量学。为了验证该系统，ANVUR进行了一项实验，以估计知情评论和文献计量学之间的协议。Ancaiani et al.（2015）介绍了实验的主要结果。Baccini和De Nicolao（2017）在一封信中记录了其他关键问题，其中包括统计分析没有在随机样本文章中实现。研究评价（Benedetto et al. 2017）发表了对该信函的回复。这份说明强调指出，答复中存在（1）数据错误，（2）样本的“代表性”问题，（3）关于计算kappas所用权重的说法无法核实，（4）未披露平均程序;（5）关于“所有领域的协议相同”的说法与官方报告相矛盾。最后但并非最不重要的是：作者使用的数据仍然没有披露。一个一般性的警告得出结论：许多最近发表的论文使用的数据来自意大利的研究评估工作。科学界无法获得这些数据，因此这些论文不可复制。至少在作者或ANVUR披露复制所需的数据之前，它们很难被认为包含可靠的证据。
The dispersion of research performance within and between universities as a potential indicator of the competitive intensity in higher education systems
