面向隐私保护的代码生成：差分隐私代码语言模型
Towards Privacy-Preserving Code Generation: Differentially Private Code Language Models
AI总结：
本文提出在代码语言模型中应用差分隐私以缓解记忆风险，同时保持代码生成能力，验证了DP在隐私保护中的有效性。
摘要：
大型专门用于代码的大型语言模型（CodeLLMs）在生成代码片段、文档和测试用例方面展现了显著的能力。然而，尽管这些模型具有巨大的潜力，但它们可能会无意中记忆并复现训练数据中的片段，这带来了隐私泄露和知识产权侵权的风险。这些风险限制了CodeLLMs在敏感领域中的部署，并限制了其训练数据集只能使用公开来源。为了解决这一记忆风险而不影响其任务性能，我们对CodeLLMs应用差分隐私（DP）。据我们所知，这是首次系统评估DP在CodeLLMs中有效性的全面研究。DP通过在训练过程中添加校准的噪声来保护个体数据点，同时仍然允许模型学习有用的模式。为此，我们首先识别并理解CodeLLMs在微调过程中的记忆行为驱动因素。然后，为了解决这一问题，我们实证评估了DP在减轻记忆的同时保持代码生成能力的效果。我们的发现表明，DP在所有测试的片段类型中显著减少了CodeLLMs的记忆。最容易记忆的片段类型也是最有效地被DP缓解的。此外，我们观察到DP略微增加了困惑度，但保留并可能增强了CodeLLMs的代码生成能力，这使得在实践中应用DP成为可能，而不会显著影响模型的实用性。最后，我们分析了DP对训练效率和能耗的影响，发现DP不会显著影响训练时间和能耗，使其成为隐私保护CodeLLMs训练的实用选择。
英文摘要：
Large language models specialized for code (CodeLLMs) have demonstrated remarkable capabilities in generating code snippets, documentation, and test cases. However, despite their promising capabilities, CodeLLMs can inadvertently memorize and reproduce snippets from their training data, which poses risks of privacy breaches and intellectual property violations. These risks restrict the deployment of CodeLLMs in sensitive domains and limit their training datasets to publicly available sources. To mitigate the memorization risk without compromising their task performance, we apply Differential Privacy (DP) to CodeLLMs. To the best of our knowledge, this is the first comprehensive study that systematically evaluates the effectiveness of DP in CodeLLMs. DP adds calibrated noise to the training process to protect individual data points while still allowing the model to learn useful patterns. To this end, we first identify and understand the driving reasons of the memorization behaviour of the CodeLLMs during their fine-tuning. Then, to address this issue, we empirically evaluate the effect of DP on mitigating memorization while preserving code generation capabilities. Our findings show that DP substantially reduces memorization in CodeLLMs across all the tested snippet types. The snippet types most prone to memorization are also the most effectively mitigated by DP. Furthermore, we observe that DP slightly increases perplexity but preserves, and can even enhance, the code generation capabilities of CodeLLMs, which makes it feasible to apply DP in practice without significantly compromising model utility. Finally, we analyze the impact of DP on training efficiency and energy consumption, finding that DP does not significantly affect training time or energy usage, making it a practical choice for privacy-preserving CodeLLMs training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
