你的神经代码补全模型使用了我的代码吗?一种成员推理方法
Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach
AI总结:
本文提出 CodeMI 成员推理方法,通过影子模型和成员分类器判断代码是否被用于训练神经代码补全模型,并揭示不同模型的成员泄漏差异。
摘要:
近年来,在开发用于自动化代码补全的深度学习模型方面取得了显著进展。尽管使用 GitHub 中的源代码训练基于深度学习的代码补全模型已成为常见做法,但这可能引发一些法律和伦理问题,例如版权侵权。在本文中,我们通过回答以下问题来调查当前神经代码补全模型的法律和伦理问题:我的代码是否被用于训练你的神经代码补全模型?为此,我们将最初为分类任务设计的成员推理方法(称为 CodeMI)调整用于更具挑战性的代码补全任务。具体而言,由于目标代码补全模型表现为不透明的黑盒,无法访问其训练数据和参数,我们选择训练多个影子模型来模拟其行为。随后,利用从这些影子模型获得的后验概率来训练一个成员分类器。接着,该成员分类器可根据目标代码补全模型的输出,有效推断给定代码样本的成员状态。我们在多种神经代码补全模型(即基于 LSTM 的模型、CodeGPT、CodeGen 和 StarCoder)上全面评估了这种适配方法的有效性。实验结果表明,基于 LSTM 的模型和 CodeGPT 模型存在成员泄漏问题,我们提出的成员推理方法可以轻松检测到该问题,准确率分别为 0.842 和 0.730。有趣的是,我们的实验还表明,当前代码大型语言模型(例如 CodeGen 和 StarCoder)的数据成员关系难以检测,这为进一步改进留下了更充足的空间。最后,我们还尝试从模型记忆的角度解释这些发现。
英文摘要:
Recent years have witnessed significant progress in developing deep learning-based models for automated code completion. Although using source code in GitHub has been a common practice for training deep-learning-based models for code completion, it may induce some legal and ethical issues such as copyright infringement. In this paper, we investigate the legal and ethical issues of current neural code completion models by answering the following question: Is my code used to train your neural code completion model? To this end, we tailor a membership inference approach (termed CodeMI) that was originally crafted for classification tasks to a more challenging task of code completion. In particular, since the target code completion models perform as opaque black boxes, preventing access to their training data and parameters, we opt to train multiple shadow models to mimic their behavior. The acquired posteriors from these shadow models are subsequently employed to train a membership classifier. Subsequently, the membership classifier can be effectively employed to deduce the membership status of a given code sample based on the output of a target code completion model. We comprehensively evaluate the effectiveness of this adapted approach across a diverse array of neural code completion models, (i.e., LSTM-based, CodeGPT, CodeGen, and StarCoder). Experimental results reveal that the LSTM-based and CodeGPT models suffer the membership leakage issue, which can be easily detected by our proposed membership inference approach with an accuracy of 0.842, and 0.730, respectively. Interestingly, our experiments also show that the data membership of current large language models of code, e.g., CodeGen and StarCoder, is difficult to detect, leaving ampler space for further improvement. Finally, we also try to explain the findings from the perspective of model memorization.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。