arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.23407cs.CRcs.AIcs.SE
收藏

通过令牌级奖励学习生成安全代码

Learning to Generate Secure Code via Token-Level Rewards

Fuyao University of Science and Technology(福耀科技大学);Xiamen University(厦门大学);Peking University(北京大学);Huawei(华为)

Jiazheng Quan, Xiaodong Li, Bin Wang, Guo An, Like Liu, Degen Huang, Lin Liu, Chengbin Hou

AI总结:

本文提出Vul2Safe和SRCode框架,通过令牌级奖励提升代码安全性,有效减少生成代码中的安全漏洞并提高代码质量。

摘要:

大型语言模型(LLMs)在代码生成方面展现了强大的能力,但它们仍然容易生成安全漏洞。现有方法通常面临两个关键限制:高质量安全数据的匮乏和粗粒度强化学习奖励信号。为了解决这些挑战,我们提出了Vul2Safe,一个新的安全代码生成框架,利用LLM自我反思来从真实世界漏洞中构建高置信度修复对,并进一步生成多样化的隐式提示以构建PrimeVul+数据集。同时,我们引入了SRCode,一种新的训练框架,首次在代码安全领域使用令牌级奖励进行强化学习,使模型在训练过程中能够持续关注并强化关键细粒度安全模式。与传统实例级奖励方案相比,我们的方法允许更精确地优化局部安全实现。大量实验表明,PrimeVul+和SRCode在生成代码中显著减少了安全漏洞,同时在多个基准上提高了整体代码质量。

英文摘要:

Large language models (LLMs) have demonstrated strong capabilities in code generation, yet they remain prone to producing security vulnerabilities. Existing approaches commonly suffer from two key limitations: the scarcity of high-quality security data and coarse-grained reinforcement learning reward signals. To address these challenges, we propose Vul2Safe, a new secure code generation framework that leverages LLM self-reflection to construct high-confidence repair pairs from real-world vulnerabilities, and further generates diverse implicit prompts to build the PrimeVul+ dataset. Meanwhile, we introduce SRCode, a novel training framework that pioneers the use of token-level rewards in reinforcement learning for code security, which enables the model to continuously attend to and reinforce critical fine-grained security patterns during training. Compared with traditional instance-level reward schemes, our approach allows for more precise optimization of local security implementations. Extensive experiments show that PrimeVul+ and SRCode substantially reduce security vulnerabilities in generated code while improving overall code quality across multiple benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑