arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.04299cs.SE
收藏

关于自动检测AI生成源代码的实证研究:我们进展如何?

An Empirical Study on Automatically Detecting AI-Generated Source Code: How Far Are We?

Hyunjae Suh, Mahan Tafreshipour, Jiawei Li, Adithya Bhattiprolu, Iftekhar Ahmed

AI总结:

本研究实证评估现有AI检测工具,发现其效果不佳;提出微调LLM及基于静态指标或AST嵌入的机器学习方法,最佳模型F1达82.55,超越GPTSniffer。

摘要:

人工智能(AI)技术,尤其是大型语言模型(LLMs),已开始在研究人员和软件开发人员中流行,用于生成源代码。然而,研究表明LLMs生成的代码存在质量问题,并引发了版权/许可侵权问题。因此,检测一段源代码是由人类还是AI编写变得必要。本研究首先进行了一项实证分析,以调查现有AI检测工具在检测AI生成代码方面的有效性。结果表明,这些工具均表现不佳,缺乏足够的泛化能力,无法实际部署。然后,为了提高AI生成代码检测的性能,我们提出了一系列方法,包括微调LLMs以及基于静态代码指标或从抽象语法树(AST)生成的代码嵌入的机器学习分类。我们最好的模型优于最先进的AI生成代码检测器(GPTSniffer),并达到了82.55的F1分数。我们还对我们表现最好的模型进行了消融研究,以调查不同源代码特征对其性能的影响。

英文摘要:

Artificial Intelligence (AI) techniques, especially Large Language Models (LLMs), have started gaining popularity among researchers and software developers for generating source code. However, LLMs have been shown to generate code with quality issues and also incurred copyright/licensing infringements. Therefore, detecting whether a piece of source code is written by humans or AI has become necessary. This study first presents an empirical analysis to investigate the effectiveness of the existing AI detection tools in detecting AI-generated code. The results show that they all perform poorly and lack sufficient generalizability to be practically deployed. Then, to improve the performance of AI-generated code detection, we propose a range of approaches, including fine-tuning the LLMs and machine learning-based classification with static code metrics or code embedding generated from Abstract Syntax Tree (AST). Our best model outperforms state-of-the-art AI-generated code detector (GPTSniffer) and achieves an F1 score of 82.55. We also conduct an ablation study on our best-performing model to investigate the impact of different source code features on its performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑