arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.10397cs.CL
收藏

大海捞针:针对摘要任务的MTurk高一致性标注者分析

Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization

New York University(纽约大学);ADAPT Centre, DCU(都柏林城市大学ADAPT中心);IBM Research(IBM研究院);Google(谷歌);Google Research(谷歌研究院);Yale University(耶鲁大学);trivago N.V.(途易集团);University of Edinburgh(爱丁堡大学);Allen Institute for AI(艾伦人工智能研究所)

Lining Zhang, Simon Mille, Yufang Hou, Daniel Deutsch, Elizabeth Clark, Yixin Liu, Saad Mahamood, Sebastian Gehrmann, Miruna Clinciu, Khyathi Chandu, João Sedoc

AI总结:

本研究针对低质量标注资源浪费问题,提出两步式流水线招募筛选MTurk高质量摘要标注者的方法,可获高一致性标注,但与专家判断一致性仍需提升,为同类标注任务招募提供实践参考。

摘要:

为避免在低质量标注上耗费高昂且低效的资源,我们需要一种方法来组建一批可靠的标注者,使其能有效完成自动摘要评估这类困难任务。为此,我们研究了通过两步式流水线招募高质量Amazon Mechanical Turk(亚马逊土耳其机器人,众包平台)标注者的方案。我们证明,能够在标注者开展评估前筛选出不合格人员,在资源约束相近的情况下获得高一致性标注。尽管我们招募的标注者之间、以及与CloudResearch标注者之间都表现出很强的共识,但他们在部分数据上与专家判断的一致性未达预期,还需在正确性方面接受进一步训练。本文仍可为其他具有挑战性的标注任务中招募合格标注者提供最佳实践参考。

英文摘要:

To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization. Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline. We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain high-agreement annotations with similar constraints on resources. Although our workers demonstrate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness. This paper still serves as a best practice for the recruitment of qualified annotators in other challenging annotation tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑