arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.09381cs.SE
收藏

这段代码片段是由 ChatGPT 编写的吗?基于 CodeBERT 分类器的实证研究

Is this Snippet Written by ChatGPT? An Empirical Study with a CodeBERT-Based Classifier

Phuong T. Nguyen, Juri Di Rocco, Claudio Di Sipio, Riccardo Rubei, Davide Di Ruscio, Massimiliano Di Penta

AI总结:

本文提出基于 CodeBERT 的 GPTSniffer 方法,用于检测 ChatGPT 生成的代码片段,并通过实证研究证明其分类准确率优于 GPTZero 和 OpenAI Text Classifier,同时分析了训练数据相似性与成对代码片段上下文对分类性能的提升作用。

摘要:

自 2022 年 11 月发布以来,ChatGPT 在用户中越来越受欢迎,尤其是程序员将其作为解决开发问题的工具。然而,尽管 ChatGPT 为编程问题提供了实用的解决方案,但它主要应作为辅助工具(例如在软件教育中)使用,而不是替代人类。因此,检测由 ChatGPT 自动生成的源代码是必要的,用于识别 AI 生成内容的工具可能需要调整以有效处理源代码。本文提出了一项实证研究,以探究自动识别 AI 生成代码片段的可行性,以及影响该能力的因素。为此,我们提出了一种名为 GPTSniffer 的新方法,该方法基于 CodeBERT 构建,用于检测由 AI 编写的源代码。结果表明,GPTSniffer 能够准确分类代码是人类编写还是 AI 生成,并且优于两个基线方法 GPTZero 和 OpenAI Text Classifier。此外,研究还表明,相似的训练数据或带有成对代码片段的分类上下文有助于提升分类性能。

英文摘要:

Since its launch in November 2022, ChatGPT has gained popularity among users, especially programmers who use it as a tool to solve development problems. However, while offering a practical solution to programming problems, ChatGPT should be mainly used as a supporting tool (e.g., in software education) rather than as a replacement for the human being. Thus, detecting automatically generated source code by ChatGPT is necessary, and tools for identifying AI-generated content may need to be adapted to work effectively with source code. This paper presents an empirical study to investigate the feasibility of automated identification of AI-generated code snippets, and the factors that influence this ability. To this end, we propose a novel approach called GPTSniffer, which builds on top of CodeBERT to detect source code written by AI. The results show that GPTSniffer can accurately classify whether code is human-written or AI-generated, and outperforms two baselines, GPTZero and OpenAI Text Classifier. Also, the study shows how similar training data or a classification context with paired snippets helps to boost classification performances.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑