arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.05534cs.SEcs.HCcs.LG
收藏

大语言模型能否替代软件工程制品的的人工标注?

Can LLMs Replace Manual Annotation of Software Engineering Artifacts?

University of California, Davis(加州大学戴维斯分校);IBM Research(IBM研究院);Singapore Management University(新加坡管理大学);University of Stuttgart(斯图加特大学)

Toufique Ahmed, Premkumar Devanbu, Christoph Treude, Michael Pradel

AI总结:

本文研究用大语言模型替代软件工程评估中昂贵的人类标注,通过在多个标注任务上验证LLM与人类评分者一致性接近,并提出模型间一致性和置信度作为选择依据,为混合评估奠定基础。

摘要:

软件工程创新(例如工具和流程)的实验评估通常将人类受试者研究作为多管齐下策略的一部分,以获得研究结果更强的普适性。然而,由于寻找和雇用合适受试者(理想情况下是具有不同经验水平的专业程序员)的成本和难度,我们领域的人类受试者研究充满挑战。与此同时,大语言模型(LLMs)最近开始在多个领域展现出人类水平的表现。本文探讨了在代码及代码相关制品的评估中,用成本低得多的LLM查询来替代昂贵的人类受试者的可能性。我们通过将六种最先进的大语言模型应用于先前工作创建的五个数据集中的十个标注任务来研究这一想法,这些任务包括判断一个方法的自然语言摘要的准确性,或决定一个代码更改是否修复了一个静态分析警告。我们的结果表明,用LLM替代部分人工标注工作可以产生与人类评分者一致性相等或接近的评分者间一致性。为了帮助决定何时以及如何在人类受试者研究中使用LLM,我们提出模型-模型一致性作为判断给定任务是否适合使用LLM的预测指标,并提出模型置信度作为选择LLM可以安全替代人类标注者的特定样本的手段。总体而言,我们的工作是迈向软件工程中人类与LLM混合评估的第一步。

英文摘要:

Experimental evaluations of software engineering innovations, e.g., tools and processes, often include human-subject studies as a component of a multi-pronged strategy to obtain greater generalizability of the findings. However, human-subject studies in our field are challenging, due to the cost and difficulty of finding and employing suitable subjects, ideally, professional programmers with varying degrees of experience. Meanwhile, large language models (LLMs) have recently started to demonstrate human-level performance in several areas. This paper explores the possibility of substituting costly human subjects with much cheaper LLM queries in evaluations of code and code-related artifacts. We study this idea by applying six state-of-the-art LLMs to ten annotation tasks from five datasets created by prior work, such as judging the accuracy of a natural language summary of a method or deciding whether a code change fixes a static analysis warning. Our results show that replacing some human annotation effort with LLMs can produce inter-rater agreements equal or close to human-rater agreement. To help decide when and how to use LLMs in human-subject studies, we propose model-model agreement as a predictor of whether a given task is suitable for LLMs at all, and model confidence as a means to select specific samples where LLMs can safely replace human annotators. Overall, our work is the first step toward mixed human-LLM evaluations in software engineering.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑