大众能客观地识别错误信息吗?判断量表与评估者背景的影响
Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background
摘要:
真实性判断是对抗错误信息过程中的一个基本步骤,因为它们对于训练和评估自动区分真假陈述的分类器至关重要。通常这种判断是由专家做出的,比如记者发表政治声明,医生发表医学声明。在本文中,我们遵循不同的方法,并依赖于(非专家)人群工作者。这当然会导致以下研究问题:众包可以可靠地用于评估信息的真实性,并创建大规模的标记收集信息可信度系统?为了解决这个问题,我们提出了基于众包的广泛研究的结果:我们收集了两个数据集上的数千个真实性评估,并将专家的判断与人群的判断进行了比较,这些判断在不同粒度级别的尺度上表示。我们还测量了工人的政治偏见和认知背景,并量化了它们对人群提供的数据可靠性的影响。
英文摘要:
Truthfulness judgments are a fundamental step in the process of fighting misinformation, as they are crucial to train and evaluate classifiers that automatically distinguish true and false statements. Usually such judgments are made by experts, like journalists for political statements or medical doctors for medical statements. In this paper, we follow a different approach and rely on (non-expert) crowd workers. This of course leads to the following research question: Can crowdsourcing be reliably used to assess the truthfulness of information and to create large-scale labeled collections for information credibility systems? To address this issue, we present the results of an extensive study based on crowdsourcing: we collect thousands of truthfulness assessments over two datasets, and we compare expert judgments with crowd judgments, expressed on scales with various granularity levels. We also measure the political bias and the cognitive background of the workers, and quantify their effect on the reliability of the data provided by the crowd.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。