人工的人工的人工智能:众包工作者在文本生成任务中广泛使用大语言模型
Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks
AI总结:
本研究针对众包工作者可能使用大语言模型完成标注任务的问题,通过击键检测和合成文本分类方法开展案例研究,发现33%-46%的众包工作者在文本生成任务中使用LLMs,呼吁各方探索保障人类数据纯粹性的新方法。
摘要:
大语言模型(LLMs)是出色的数据标注工具,可用于生成高保真监督训练数据以及调查和实验数据。随着LLMs的广泛应用,人类金标准标注是理解LLMs能力及其结果有效性的关键。然而,众包作为获取人类标注的重要且低成本的方式,本身可能受到LLMs的影响,因为众包工作者有使用LLMs来提高生产力和收入的经济动机。为调查这一问题,我们开展了一项关于众包工作者使用LLMs普遍性的案例研究。我们在Amazon Mechanical Turk平台上复现了文献中的一项摘要总结任务,通过击键检测与合成文本分类相结合的方法,估计有33%-46%的众包工作者在完成任务时使用了LLMs。尽管目前尚不清楚该结论能否推广到其他对LLMs友好度较低的任务中,但我们的研究结果呼吁平台、研究者与众包工作者寻找新方法来确保人类数据的纯人类属性,或许可将本文提出的方法作为起点。代码与数据见:https://github.com/epfl-dlab/GPTurk
英文摘要:
Large language models (LLMs) are remarkable data annotators. They can be used to generate high-fidelity supervised training data, as well as survey and experimental data. With the widespread adoption of LLMs, human gold--standard annotations are key to understanding the capabilities of LLMs and the validity of their results. However, crowdsourcing, an important, inexpensive way to obtain human annotations, may itself be impacted by LLMs, as crowd workers have financial incentives to use LLMs to increase their productivity and income. To investigate this concern, we conducted a case study on the prevalence of LLM usage by crowd workers. We reran an abstract summarization task from the literature on Amazon Mechanical Turk and, through a combination of keystroke detection and synthetic text classification, estimate that 33-46% of crowd workers used LLMs when completing the task. Although generalization to other, less LLM-friendly tasks is unclear, our results call for platforms, researchers, and crowd workers to find new ways to ensure that human data remain human, perhaps using the methodology proposed here as a stepping stone. Code/data: https://github.com/epfl-dlab/GPTurk
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。