置信度校准：小组评估的原则性方法
Calibration with confidence: A principled method for panel assessment
摘要：
通常，一组对象必须由一个评估小组评估，但并非每个对象都由每个评估员评估。这些小组面临的一个问题是如何考虑小组成员之间的不同标准和对他们的分数的不同置信水平。在这里，开发了一种基于几何的算法来校准这些评估员的分数，解决了这两个问题。该算法是基于评估员和对象的图的连接性评估，将其边缘上的权重声明的信心。如果图表连接得足够好，则可以通过比较评估员如何对他们共同评估的对象进行评级来推断相对标准，并通过每个评估的置信水平进行加权。通过消除这些偏差，可以为所有对象推断出“真”值。得到的值的可靠性估计。该算法进行了测试，在两个案例研究中，一个通过计算机模拟，另一个基于现实的评估数据。这个过程是比较简单的平均过程中广泛使用，并Fisher的添加剂不完全区组分析。预计该算法将在各种情况下证明是有用的，如评价提交给国家评估工作的研究质量;评估提交给资助小组的赠款建议;职位申请人的排名;以及判断候选人可以从选项列表中选择的学位课程的表现。
英文摘要：
Frequently, a set of objects has to be evaluated by a panel of assessors, but not every object is assessed by every assessor. A problem facing such panels is how to take into account different standards amongst panel members and varying levels of confidence in their scores. Here, a mathematically-based algorithm is developed to calibrate the scores of such assessors, addressing both of these issues. The algorithm is based on the connectivity of the graph of assessors and objects evaluated, incorporating declared confidences as weights on its edges. If the graph is sufficiently well connected, relative standards can be inferred by comparing how assessors rate objects they assess in common, weighted by the levels of confidence of each assessment. By removing these biases, "true" values are inferred for all the objects. Reliability estimates for the resulting values are obtained. The algorithm is tested in two case studies, one by computer simulation and another based on realistic evaluation data. The process is compared to the simple averaging procedure in widespread use, and to Fisher's additive incomplete block analysis. It is anticipated that the algorithm will prove useful in a wide variety of situations such as evaluation of the quality of research submitted to national assessment exercises; appraisal of grant proposals submitted to funding panels; ranking of job applicants; and judgement of performances on degree courses wherein candidates can choose from lists of options.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
