arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.06835cs.CL
收藏

使用Mechanical Turk评估开放式文本生成的风险

The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation

University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校)

Marzena Karpinska, Nader Akoury, Mohit Iyyer

摘要:

最近的文本生成研究越来越多地集中在开放的领域,如故事和诗歌的生成。由于为这些任务构建的模型难以自动评估,该领域的大多数研究人员通过收集众包的文本质量人类判断来证明他们的建模选择(例如,亚马逊土耳其机器人(AMT)的连贯性或语法性的李克特分数)。在本文中,我们首先对45篇开放式文本生成论文进行了调查,发现其中绝大多数未能报告有关AMT任务的关键细节,从而阻碍了可重复性。然后,我们与AMT工人和英语教师进行了一系列的故事评估实验,发现即使有严格的资格过滤器,AMT工人(不像教师)也无法区分模型生成的文本和人类生成的参考。我们发现,AMT工人的判断,提高当他们显示模型生成的输出与人类生成的参考,这使得工人更好地校准他们的评级。最后,与英语教师的访谈提供了更深入的见解,评估过程中的挑战,特别是当评级模型生成的文本。

英文摘要:

Recent text generation research has increasingly focused on open-ended domains such as story and poetry generation. Because models built for such tasks are difficult to evaluate automatically, most researchers in the space justify their modeling choices by collecting crowdsourced human judgments of text quality (e.g., Likert scores of coherence or grammaticality) from Amazon Mechanical Turk (AMT). In this paper, we first conduct a survey of 45 open-ended text generation papers and find that the vast majority of them fail to report crucial details about their AMT tasks, hindering reproducibility. We then run a series of story evaluation experiments with both AMT workers and English teachers and discover that even with strict qualification filters, AMT workers (unlike teachers) fail to distinguish between model-generated text and human-generated references. We show that AMT worker judgments improve when they are shown model-generated output alongside human-generated references, which enables the workers to better calibrate their ratings. Finally, interviews with the English teachers provide deeper insights into the challenges of the evaluation process, particularly when rating model-generated text.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑