无先验地引出思维层级
Eliciting Thinking Hierarchy without a Prior
AI总结:
本文研究在无先验条件下从群体回答中识别更资深人群的正确答案,提出思维层级模型、学习算法及开放式众包方法,并通过实证表明其准确率显著优于多数投票。
摘要:
当我们利用群体智慧时,通常根据答案的流行度进行排序,尤其是在无法验证答案的情况下。然而,当多数人犯系统性错误时,这种做法可能非常危险。一个根本性问题随之而来:我们能否在没有任何先验的情况下,在答案之间构建一个层级,使排名更高的答案(可能并不被多数人支持)来自更资深的人?为了解决这一问题,我们提出:1)一种描述人们思维层级的新模型;2)两种在没有任何先验的情况下学习思维层级的算法;3)一种基于上述理论框架的新型开放式回答众包方法。除理论论证外,我们还进行了四项实证众包研究,并表明:a)我们的方法学习到的排名最高答案的准确率远高于多数投票(在一个问题中,多数答案得到74名受访者支持,但正确答案仅得到3名受访者支持。我们的方法在没有任何先验的情况下将正确答案排在最高位);b)我们的模型具有很高的拟合优度,尤其是在排名最高答案为正确的问题上。据我们所知,我们是第一个在一般问题求解场景中提出具有实证验证的思维层级模型的研究;也是第一个提出实用的基于开放式回答的众包方法,在没有任何先验的情况下优于多数投票。
英文摘要:
When we use the wisdom of the crowds, we usually rank the answers according to their popularity, especially when we cannot verify the answers. However, this can be very dangerous when the majority make systematic mistakes. A fundamental question arises: can we build a hierarchy among the answers \textit{without any prior} where the higher-ranking answers, which may not be supported by the majority, are from more sophisticated people? To address the question, we propose 1) a novel model to describe people's thinking hierarchy; 2) two algorithms to learn the thinking hierarchy without any prior; 3) a novel open-response based crowdsourcing approach based on the above theoretic framework. In addition to theoretic justifications, we conduct four empirical crowdsourcing studies and show that a) the accuracy of the top-ranking answers learned by our approach is much higher than that of plurality voting (In one question, the plurality answer is supported by 74 respondents but the correct answer is only supported by 3 respondents. Our approach ranks the correct answer the highest without any prior); b) our model has a high goodness-of-fit, especially for the questions where our top-ranking answer is correct. To the best of our knowledge, we are the first to propose a thinking hierarchy model with empirical validations in the general problem-solving scenarios; and the first to propose a practical open-response based crowdsourcing approach that beats plurality voting without any prior.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。