arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.14766cs.CRcs.AI
收藏

最后一个安全代码基准:你“ vibe 编码”的代码安全吗?

Last Secure Code Benchmark: Is the Code You Vibe Secure?

National University of Singapore(新加坡国立大学);Qwen Team, Alibaba Token Hub, Alibaba Group(阿里通义千问团队、阿里Token Hub、阿里巴巴集团);Cloud Safety and Security, Alibaba Cloud, Alibaba Group(阿里云云安全与安全部门、阿里云、阿里巴巴集团)
Tianyi Wu, Chao Ge, Peng Chen, Fengji Zhang, Yongxing Wu, Yuchong Xie, Jian Li, Chaoyuan Peng, Kaikai Zhang, Dong Huang, Mingzhe Du, Yukai Huang, Jiaheng Zhang,… 展开作者

Tianyi Wu, Chao Ge, Peng Chen, Fengji Zhang, Yongxing Wu, Yuchong Xie, Jian Li, Chaoyuan Peng, Kaikai Zhang, Dong Huang, Mingzhe Du, Yukai Huang, Jiaheng Zhang, Terry Yue Zhuo, See-Kiong Ng

AI总结:

本研究推出 Last Secure Code Benchmark 评估编码智能体的安全代码生成,发现前沿智能体生成的代码功能通过率高但安全通过率低,vibe 编码生成代码的安全性存在严重隐患。

摘要:

AI 编码智能体现在编写了大量已交付的代码,但这些代码是否安全仍不明确。我们推出了 Last Secure Code Benchmark,这是一项针对编码智能体安全代码生成的综合评估。该基准基于 150 个真实漏洞构建,包含 450 个任务,覆盖 7 种广泛使用的语言(Java、C、C++、JavaScript、Python、PHP 和 Go)、24 种 CWE(其中包括 2025 年 CWE 前 25 名中的 21 种以及前 10 种 KEV 弱点中的 8 种),数量超过我们对比的任何基准,还包含 3 种 vibe 编码场景:补全函数、编写文件以及从零构建仓库的源代码树。每个任务仅通过规范要求功能,从未提及安全性,结果通过隐藏的功能测试和安全测试进行评分。在 6 个前沿智能体中,73.6% 至 91.3% 的任务通过功能测试,但即使是最强的 Claude Fable 5,也仅在 25.1% 的任务中生成既功能正常又安全的代码,其他智能体的这一比例为 16.4% 至 24.7%。最佳模型在所有 vibe 编码场景的漏洞场景中仅通过 12%,这对 vibe 编码生成代码的安全性构成了严重担忧。

英文摘要:

AI coding agents now write much of the code that ships, yet whether that code is secure remains unclear. We introduce the Last Secure Code Benchmark, a comprehensive evaluation of secure code generation by coding agents. Built from 150 real vulnerabilities, its 450 tasks span seven widely used languages (Java, C, C++, JavaScript, Python, PHP, and Go), 24 CWEs including 21 of the 2025 CWE Top 25 and 8 of the Top 10 KEV weaknesses, more than any benchmark we compare with, and three vibe-coding scenarios: filling in a function, writing a file, and building a repository's source tree from scratch. Every task asks only for functionality, through a specification that never mentions security, and hidden functional and security tests grade the result. Across six frontier agents, 73.6% to 91.3% of tasks pass functionally, yet even the strongest, Claude Fable 5, produces code that is both functional and secure for only 25.1% of tasks, and the others for 16.4% to 24.7%. The best model only passes 12% of the vulnerability scenarios across all vibe coding setting, posing strong concern on the security of the code written by vibe coding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑