揭示大语言模型中的预训练代码:一种语法感知的归因方法
Uncovering Pretraining Code in LLMs: A Syntax-Aware Attribution Approach
AI总结:
本文提出 SynPrune,一种利用代码语法结构进行剪枝的成员推断攻击方法,通过排除语法必需但不反映作者风格的 token 来检测代码是否被用于 LLM 预训练,实验证明其性能优于现有方法且具有鲁棒性。
摘要:
随着大语言模型(LLMs)能力日益增强,人们对其训练数据中未经授权使用受版权保护和许可内容的担忧不断加剧,尤其是在代码领域。开源代码通常受开源许可证(如 GPL)保护,在预训练中使用时会引发法律和伦理挑战。因此,检测特定代码样本是否被包含在 LLM 训练数据中,对于透明度、问责制和版权合规至关重要。我们提出 SynPrune,一种针对代码的语法剪枝成员推断攻击方法。与以往将代码视为纯文本的 MIA 方法不同,SynPrune 利用编程语言的结构化和规则约束特性。具体而言,该方法在计算成员分数时,识别并排除那些在语法上必需、且不能反映作者风格的后续 token,使其不参与归因。实验结果表明,SynPrune 持续优于当前最先进的方法。我们的方法在不同函数长度和语法类别下也表现出鲁棒性。
英文摘要:
As large language models (LLMs) become increasingly capable, concerns over the unauthorized use of copyrighted and licensed content in their training data have grown, especially in the context of code. Open-source code, often protected by open source licenses (e.g, GPL), poses legal and ethical challenges when used in pretraining. Detecting whether specific code samples were included in LLM training data is thus critical for transparency, accountability, and copyright compliance. We propose SynPrune, a syntax-pruned membership inference attack method tailored for code. Unlike prior MIA approaches that treat code as plain text, SynPrune leverages the structured and rule-governed nature of programming languages. Specifically, it identifies and excludes consequent tokens that are syntactically required and not reflective of authorship, from attribution when computing membership scores. Experimental results show that SynPrune consistently outperforms the state-of-the-arts. Our method is also robust across varying function lengths and syntax categories.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。