arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1406.6618stat.MLcs.LG
收藏

什么时候比较比评分更好?

When is it Better to Compare than to Score?

University of California, Berkeley(加州大学伯克利分校)

Nihar B. Shah, Sivaraman Balakrishnan, Joseph Bradley, Abhay Parekh, Kannan Ramchandran, Martin Wainwright

摘要:

当人类对未知量做出判断时,人们通常可以选择直接评分(基数)或比较(序数)测量。在本文中,我们研究了这两种选择的相对优点,为测量方案的选择提供了经验和理论指导。我们提供的经验证据的基础上亚马逊土耳其机械的实验,在各种任务中,(成对比较)有序测量具有较低的每个样本的噪音,通常比基数更快地引出。然而,顺序测量通常提供较少的信息。然后,我们考虑流行的Thurstone和Bradley-Terry-Luce(BTL)模型的顺序测量和特征的极小极大误差率估计未知量。我们比较这些最小最大误差率的基数测量模型和量化的噪声水平,有序测量更好。最后,我们重新审视了从我们的实验中收集的数据,并表明拟合这些模型证实了这一预测:对于有序测量中的噪声足够低的任务,有序方法导致估计误差较小。

英文摘要:

When eliciting judgements from humans for an unknown quantity, one often has the choice of making direct-scoring (cardinal) or comparative (ordinal) measurements. In this paper we study the relative merits of either choice, providing empirical and theoretical guidelines for the selection of a measurement scheme. We provide empirical evidence based on experiments on Amazon Mechanical Turk that in a variety of tasks, (pairwise-comparative) ordinal measurements have lower per sample noise and are typically faster to elicit than cardinal ones. Ordinal measurements however typically provide less information. We then consider the popular Thurstone and Bradley-Terry-Luce (BTL) models for ordinal measurements and characterize the minimax error rates for estimating the unknown quantity. We compare these minimax error rates to those under cardinal measurement models and quantify for what noise levels ordinal measurements are better. Finally, we revisit the data collected from our experiments and show that fitting these models confirms this prediction: for tasks where the noise in ordinal measurements is sufficiently low, the ordinal approach results in smaller errors in the estimation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑