arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.07091cs.SE
收藏

基于LLM的评审评论生成在实践中的影响:一项混合开源/闭源用户研究

Impact of LLM-based Review Comment Generation in Practice: A Mixed Open-/Closed-source User Study

Doriane Olewicki, Leuson Da Silva, Suhaib Mujahid, Arezou Amini, Benjamin Mah, Marco Castelluccio, Sarra Habchi, Foutse Khomh, Bram Adams

AI总结:

本研究通过Mozilla和Ubisoft的实证用户研究,评估了基于LLM的评审评论生成工具RevMate的实际影响,发现其评论接受率较低但有一定价值,且不显著增加评审时间负担。

摘要:

我们在实际运行环境中开展了一项大规模实证用户研究,以评估LLM生成评论的接受度及其对评审过程的影响。该用户研究在两个组织中进行:Mozilla(其代码库为开源)和Ubisoft(完全闭源)。在参与者通常使用的评审环境中,他们可以使用RevMate——一个基于LLM的辅助工具,该工具使用现成的LLM结合检索增强生成来提供额外的代码和评审上下文,并建议生成的评审评论,同时结合LLM-as-a-Judge自动评估生成的评论并丢弃不相关的案例。基于RevMate提供的超过587个补丁评审,我们观察到两个组织中分别有8.1%和7.2%的LLM生成评论被评审者接受,而另外14.6%和20.5%的评论仍被视为有价值的评审或开发提示。与重构相关的评论比功能性评论更可能被接受(分别为18.2%和18.6%,而功能性评论为4.8%和5.2%)。评审者检查生成的评论或编辑已接受的评论(36/119)所花费的额外时间合理,每个补丁的总体中位数为43秒。被接受的生成评论与人工编写的评论一样有可能导致修订后补丁的未来修订(在块级别为74%对73%)。

英文摘要:

We conduct a large-scale empirical user study in a live setup to evaluate the acceptance of LLM-generated comments and their impact on the review process. This user study was performed in two organizations, Mozilla (which has its codebase available as open source) and Ubisoft (fully closed-source). Inside their usual review environment, participants were given access to RevMate, an LLM-based assistive tool suggesting generated review comments using an off-the-shelf LLM with Retrieval Augmented Generation to provide extra code and review context, combined with LLM-as-a-Judge, to auto-evaluate the generated comments and discard irrelevant cases. Based on more than 587 patch reviews provided by RevMate, we observed that 8.1% and 7.2%, respectively, of LLM-generated comments were accepted by reviewers in each organization, while 14.6% and 20.5% other comments were still marked as valuable as review or development tips. Refactoring-related comments are more likely to be accepted than Functional comments (18.2% and 18.6% compared to 4.8% and 5.2%). The extra time spent by reviewers to inspect generated comments or edit accepted ones (36/119), yielding an overall median of 43s per patch, is reasonable. The accepted generated comments are as likely to yield future revisions of the revised patch as human-written comments (74% vs 73% at chunk-level).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑