真实性的多个维度:多维规模的众包错误信息评估
The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale
摘要:
最近的工作表明,利用众包作为评估公开声明真实性的工具是可行的。在某些条件下,例如:(1)拥有一组具有不同背景和认知能力的平衡的工作人员;(2)使用一套适当的机制来控制收集的数据的质量;(3)使用粗粒度的评估量表,人群可以提供可靠的假新闻识别。然而,假新闻是一个微妙的问题:声明可能只是有偏见(“cherrypicked”),不精确,错误等,现有工作中使用的一维真理量表无法解释这些差异。在本文中,我们提出了一个多维的真实性的概念,我们要求群众工作者评估七个不同的维度的真实性选择基于现有的文献:正确性,中立性,可理解性,精确性,完整性,扬声器的可信度,和信息。我们部署了一套质量控制机制,以确保分布在两个数据集上的180个公开的事实核查声明中收集的数千个评估具有足够的质量,包括人群工作人员用来查找支持其真实性评估的网页的自定义搜索引擎。对众包判断的全面分析表明:(1)与专家提供的黄金标准相比,众包评估是可靠的;(2)真实性的拟议维度捕获独立的信息;(3)众包任务可以很容易地被工作人员学习;(4)由此产生的评估为更完整的评估陈述真实性提供了有用的基础。
英文摘要:
Recent work has demonstrated the viability of using crowdsourcing as a tool for evaluating the truthfulness of public statements. Under certain conditions such as: (1) having a balanced set of workers with different backgrounds and cognitive abilities; (2) using an adequate set of mechanisms to control the quality of the collected data; and (3) using a coarse grained assessment scale, the crowd can provide reliable identification of fake news. However, fake news are a subtle matter: statements can be just biased ("cherrypicked"), imprecise, wrong, etc. and the unidimensional truth scale used in existing work cannot account for such differences. In this paper we propose a multidimensional notion of truthfulness and we ask the crowd workers to assess seven different dimensions of truthfulness selected based on existing literature: Correctness, Neutrality, Comprehensibility, Precision, Completeness, Speaker's Trustworthiness, and Informativeness. We deploy a set of quality control mechanisms to ensure that the thousands of assessments collected on 180 publicly available fact-checked statements distributed over two datasets are of adequate quality, including a custom search engine used by the crowd workers to find web pages supporting their truthfulness assessments. A comprehensive analysis of crowdsourced judgments shows that: (1) the crowdsourced assessments are reliable when compared to an expert-provided gold standard; (2) the proposed dimensions of truthfulness capture independent pieces of information; (3) the crowdsourcing task can be easily learned by the workers; and (4) the resulting assessments provide a useful basis for a more complete estimation of statement truthfulness.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。