arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.16133cs.SEcs.AI
收藏

揭示LLM生成的代码:一种基于代码重写的零样本合成代码检测器

Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting

Zhejiang University(浙江大学);Stony Brook University(石溪大学)

Tong Ye, Yangkai Du, Tengfei Ma, Lingfei Wu, Xuhong Zhang, Shouling Ji, Wenhai Wang

AI总结:

本文提出一种基于代码重写相似性的零样本合成代码检测器,利用自监督对比学习训练相似度模型,在APPS和MBPP基准上AUROC分别提升20.5%和29.1%。

摘要:

大型语言模型(LLMs)在生成代码方面展现出了卓越的能力。然而,LLM生成的(合成)代码的滥用已在教育和工业领域引发担忧,凸显了对合成代码检测器的迫切需求。现有的合成内容检测方法主要针对通用文本设计,由于编程语言独特的语法结构以及大量“低熵”标记的存在,这些方法在处理代码时效果不佳。基于此,我们的工作提出了一种新颖的零样本合成代码检测器,其依据是原始代码与其LLM重写变体之间的相似性。我们的方法基于以下观察:当原始代码是合成代码时,LLM重写版本与原始代码之间的差异往往更小。我们利用自监督对比学习来训练一个代码相似度模型,并在两个合成代码检测基准上评估了我们的方法。结果表明,与现有的最先进(SOTA)合成内容检测器相比,我们的方法取得了显著改进,在APPS基准上AUROC分数提高了20.5%,在MBPP基准上提高了29.1%。

英文摘要:

Large Language Models (LLMs) have demonstrated remarkable proficiency in generating code. However, the misuse of LLM-generated (synthetic) code has raised concerns in both educational and industrial contexts, underscoring the urgent need for synthetic code detectors. Existing methods for detecting synthetic content are primarily designed for general text and struggle with code due to the unique grammatical structure of programming languages and the presence of numerous ''low-entropy'' tokens. Building on this, our work proposes a novel zero-shot synthetic code detector based on the similarity between the original code and its LLM-rewritten variants. Our method is based on the observation that differences between LLM-rewritten and original code tend to be smaller when the original code is synthetic. We utilize self-supervised contrastive learning to train a code similarity model and evaluate our approach on two synthetic code detection benchmarks. Our results demonstrate a significant improvement over existing SOTA synthetic content detectors, with AUROC scores increasing by 20.5% on the APPS benchmark and 29.1% on the MBPP benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑