arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.01607stat.MLcs.GTcs.ITcs.LGmath.IT
收藏

序贯评估中的偏差建模与校正

Modeling and Correcting Bias in Sequential Evaluation

Georgia Institute of Technology(佐治亚理工学院)

Jingyan Wang, Ashwin Pananjady

AI总结:

针对序贯评估中的顺序偏差,提出自然模型并通过众包实验验证,设计近线性时间在线校正算法,证明其在两个排序指标上的最优性,且实验表明优于现有方法。

摘要:

我们考虑序贯评估问题,在该问题中,评估者按顺序观察候选人,并以在线、不可撤销的方式为这些候选人打分。受研究此类情境中序贯偏差(即评估结果与候选人出现顺序之间的依赖性)的心理学文献启发,我们为评估者的评分过程提出了一个自然模型,该模型捕捉了此类任务固有的校准缺失。我们开展众包实验以展示模型的多个方面。接着,我们通过将该问题表述为统计推断问题,研究如何在我们的模型下校正序贯偏差。我们为此任务提出一种近线性时间的在线算法,并针对两个经典排序指标证明其保证。我们还通过在这两个指标上建立匹配的下界,证明我们的算法在信息论意义上是最优的。最后,我们进行大量数值实验,表明无论是在模拟中还是在我们收集的众包数据上,我们的算法通常优于使用报告分数所产生排序的现行方法。

英文摘要:

We consider the problem of sequential evaluation, in which an evaluator observes candidates in a sequence and assigns scores to these candidates in an online, irrevocable fashion. Motivated by the psychology literature that has studied sequential bias in such settings -- namely, dependencies between the evaluation outcome and the order in which the candidates appear -- we propose a natural model for the evaluator's rating process that captures the lack of calibration inherent to such a task. We conduct crowdsourcing experiments to demonstrate various facets of our model. We then proceed to study how to correct sequential bias under our model by posing this as a statistical inference problem. We propose a near-linear time, online algorithm for this task and prove guarantees in terms of two canonical ranking metrics. We also prove that our algorithm is information theoretically optimal, by establishing matching lower bounds in both metrics. Finally, we perform a host of numerical experiments to show that our algorithm often outperforms the de facto method of using the rankings induced by the reported scores, both in simulation and on the crowdsourcing data that we collected.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑