SecCodeBench-V2 技术报告
SecCodeBench-V2 Technical Report
AI总结:
SecCodeBench-V2 通过动态执行和LLM-as-a-judge oracle评估AI编程助手的安全能力,提供可复现的基准测试框架。
摘要:
我们介绍了SecCodeBench-V2,一个公开发布的基准,用于评估大型语言模型(LLM)协作者生成安全代码的能力。SecCodeBench-V2 包含98个生成和修复场景,这些场景源自阿里巴巴集团的工业生产,其中底层的安全问题覆盖了22种常见的CWE(常见弱点枚举)类别,涉及五种编程语言:Java、C、Python、Go和JavaScript。SecCodeBench-V2 采用函数级任务制定:每个场景提供完整的项目框架,并要求模型在固定接口和依赖项下实现或修补指定的目标函数。对于每个场景,SecCodeBench-V2 提供可执行的证明概念(PoC)测试用例,用于功能验证和安全验证。所有测试用例均由安全专家编写并双审,确保了高保真度、广泛覆盖和可靠的地面真相。除了基准本身,我们构建了一个统一的评估流程,主要通过动态执行来评估模型。对于大多数场景,我们将在隔离环境中编译并运行模型生成的成果,并执行PoC测试用例以验证功能正确性和安全属性。对于无法通过确定性测试用例裁定安全问题的场景,我们还额外使用LLM-as-a-judge oracle。为了总结跨异构场景和难度级别的性能,我们设计了一种基于Pass@K的评分协议,通过在场景和严重性上的原则性聚合,使模型之间的评估变得全面且可比较。总体而言,SecCodeBench-V2 为评估AI编程助手的安全状况提供了严格且可重复的基础,结果和成果已发布在https://alibaba.github.io/sec-code-bench。该基准已公开在https://github.com/alibaba/sec-code-bench。
英文摘要:
We introduce SecCodeBench-V2, a publicly released benchmark for evaluating Large Language Model (LLM) copilots' capabilities of generating secure code. SecCodeBench-V2 comprises 98 generation and fix scenarios derived from Alibaba Group's industrial productions, where the underlying security issues span 22 common CWE (Common Weakness Enumeration) categories across five programming languages: Java, C, Python, Go, and JavaScript. SecCodeBench-V2 adopts a function-level task formulation: each scenario provides a complete project scaffold and requires the model to implement or patch a designated target function under fixed interfaces and dependencies. For each scenario, SecCodeBench-V2 provides executable proof-of-concept (PoC) test cases for both functional validation and security verification. All test cases are authored and double-reviewed by security experts, ensuring high fidelity, broad coverage, and reliable ground truth. Beyond the benchmark itself, we build a unified evaluation pipeline that assesses models primarily via dynamic execution. For most scenarios, we compile and run model-generated artifacts in isolated environments and execute PoC test cases to validate both functional correctness and security properties. For scenarios where security issues cannot be adjudicated with deterministic test cases, we additionally employ an LLM-as-a-judge oracle. To summarize performance across heterogeneous scenarios and difficulty levels, we design a Pass@K-based scoring protocol with principled aggregation over scenarios and severity, enabling holistic and comparable evaluation across models. Overall, SecCodeBench-V2 provides a rigorous and reproducible foundation for assessing the security posture of AI coding assistants, with results and artifacts released at https://alibaba.github.io/sec-code-bench. The benchmark is publicly available at https://github.com/alibaba/sec-code-bench.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。