抽查等价性:一种用于信息获取机制的可解释度量
Spot Check Equivalence: an Interpretable Metric for Information Elicitation Mechanisms
AI总结:
本文针对众包信息获取机制中不同性能度量结果矛盾的问题,提出“抽查等价性”这一可解释度量,统一了不同情境下的比较框架,并通过模拟实验验证了其有效性。
摘要:
由于高质量数据对人工智能系统而言如同氧气一般重要,有效从众包工作者处获取信息已成为开发高性能机器学习算法的首要问题。抽查(spot-checking)和同伴预测(peer prediction)这两种主流范式使得设计机制以评估和激励人类标注者提供高质量数据成为可能。迄今为止,至少已有三种度量被提出用于比较这些技术的性能[33, 8, 3]。然而,不同的度量在不同情境下会导致分歧甚至相互矛盾的结果。在本文中,我们调和了这些分歧的叙述,表明其中两种度量在某些情境下实际上是相同的,并解释了第三种度量产生分歧的原因。此外,我们通过引入“抽查等价性”(Spot Check Equivalence)来统一这些不同情境,该概念为同伴预测机制的有效性提供了一种可解释的度量。最后,我们提出了两种在不同情境下计算抽查等价性的方法,其中模拟结果验证了我们所提度量的有效性。
英文摘要:
Because high-quality data is like oxygen for AI systems, effectively eliciting information from crowdsourcing workers has become a first-order problem for developing high-performance machine learning algorithms. Two prevalent paradigms, spot-checking and peer prediction, enable the design of mechanisms to evaluate and incentivize high-quality data from human labelers. So far, at least three metrics have been proposed to compare the performances of these techniques [33, 8, 3]. However, different metrics lead to divergent and even contradictory results in various contexts. In this paper, we harmonize these divergent stories, showing that two of these metrics are actually the same within certain contexts and explain the divergence of the third. Moreover, we unify these different contexts by introducing \textit{Spot Check Equivalence}, which offers an interpretable metric for the effectiveness of a peer prediction mechanism. Finally, we present two approaches to compute spot check equivalence in various contexts, where simulation results verify the effectiveness of our proposed metric.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。