用于代码溯源的开源情报以及区分人类和大语言模型对常见编程任务实现的安全模式
Open-Source Intelligence for Code Provenance and the Security Patterns that Separate Human and Large-Language-Model Implementations of Common Programming Tasks
浏览论文内容
中文总结 AI 辅助
研究开发者代码来源（人类答案和大语言模型输出）的差异，利用开源构建管道收集样本，训练分类器区分溯源，报告安全差异，通过案例研究发现修复情况及问题，管道数据驱动可添加新任务或语言。
中文摘要 AI 辅助
开发者现在从两个截然不同的来源获取代码，如Stack Overflow上积累的人类答案以及大语言模型的输出。本文提出两个问题：一是能否从代码本身恢复代码片段的溯源，二是这两个来源在执行相同任务时所采用的安全模式是否不同。利用开源资源、开放权重语言模型的公共网关和公共Stack Overflow API，构建了一个完全可重现的管道，收集了9种语言模型和人类答案中31个安全敏感编程任务的真实实现，并使用确定性安全和风格检测器对每个样本进行评分。在528个真实样本上训练了交叉验证分类器，用于区分人类和模型溯源，准确率达93%，基线为78%；还训练了7路分类器，将样本归因于编写它的特定模型，准确率为48%。然后报告了两个来源在安全方面的差异，哪些模式模型比人类语料库更常采用，哪些是从人类语料库继承的。在Python、JavaScript、Go和Java中运行相同任务，发现安全差异在每种语言中都存在，而溯源边界部分特定于语言，且在它们之间不对称转移。通过一个漏洞修复案例研究，发现模型修复不安全代码的成功率为77%，但存在反复的部分修复失败情况，即模型移除了不安全模式却未添加正确防御。该管道由数据驱动，任何新任务或语言都作为单个规范条目添加，并且一个故障关闭检查器可从存储的数据中重新得出本文中的每个数字。
英文摘要
Developers now draw code from two very different sources, the accumulated human answers on sites such as Stack Overflow and the output of large language models. We ask two questions about that split. First, can the provenance of a code snippet be recovered from the code itself, and second, do the two sources differ in the security patterns they adopt for the same task. Using only open sources, a public gateway of open-weight language models and the public Stack Overflow API, we build a fully reproducible pipeline that collects real implementations of 31 security-sensitive programming tasks, among them OAuth with PKCE, JWT verification, password hashing, and SQL access, from 9 language models and from human answers, and scores every sample with deterministic security and style detectors. On 528 real samples we train a cross-validated classifier that recovers human versus model provenance with 93 percent accuracy against a 78 percent baseline, and a 7-way classifier that attributes a sample to the specific model that wrote it at 48 percent. We then report where the sources diverge on security, which patterns models adopt more often than the human corpus and which they inherit from it. Running the same tasks in Python, JavaScript, Go, and Java, we find the security divergence holds in every language while the provenance boundary is partly language-specific and does not transfer symmetrically between them. A vulnerability repair case study, in which models are handed insecure code and asked to fix it, finds a 77 percent repair rate across 21 seeds and 12 weakness classes, but a recurring partial-fix failure in which the model removes the insecure pattern without adding the correct defense. The pipeline is data driven, so any new task or language is added as a single specification entry, and a fail-closed checker re-derives every number in this paper from the stored data.
