AutoBaxBuilder: 通过代码安全基准测试进行代码安全性评估
AutoBaxBuilder: Bootstrapping Code Security Benchmarking
AI总结：
本文提出AutoBaxBuilder，一种自动化生成代码安全基准测试任务的流水线，通过结合LLM的代码理解能力与可靠性检查，构建功能测试和端到端的安全性探测利用，从而提高代码安全性的评估效率和准确性。
摘要：
随着大型语言模型（LLMs）在软件工程中的广泛应用，对LLM生成代码的正确性和安全性的可靠评估至关重要。值得注意的是，先前的研究表明LLMs容易生成包含安全漏洞的代码，凸显了安全问题常被忽视。这些见解是通过安全专家通过大量手动工作专门设计的基准测试实现的。然而，基准测试（i）不可避免地会污染训练数据，（ii）必须扩展到新任务以提供更全面的视图，（iii）必须增加难度以挑战更强大的LLMs。在本工作中，我们解决了这些挑战，并提出了AutoBaxBuilder，一种自动化流水线，能够从头开始生成代码安全基准测试任务。它利用LLM的代码理解能力，结合稳健的可靠性检查，构建功能测试和端到端的安全性探测利用。该流水线的质量通过将其预测与专家编写的基础线对齐，并通过手动验证其正确性进行定性验证。我们使用AutoBaxBuilder构建了一个新的基准测试，并将其发布给公众作为AutoBaxBench，同时对当前的LLMs进行了全面评估。AutoBaxBuilder在不到2小时内生成新的任务，费用低于4美元。包括手动验证，这将基准测试构建所需的人力工作减少了一个因素12。
英文摘要：
As large language models (LLMs) see wide adoption in software engineering, the reliable assessment of the correctness and security of LLM-generated code is crucial. Notably, prior work showed that LLMs are prone to generating code with security vulnerabilities, highlighting that security is often overlooked. These insights were enabled by specialized benchmarks crafted by security experts through significant manual effort. However, benchmarks (i) inevitably end up contaminating training data, (ii) must extend to new tasks to provide a more complete picture, and (iii) must increase in difficulty to challenge more capable LLMs. In this work, we address these challenges and present AutoBaxBuilder, an automated pipeline that generates code security benchmarking tasks from scratch. It leverages the code-understanding capabilities of LLMs combined with robust reliability checks to construct functional tests and end-to-end security-probing exploits. The quality of the pipeline is quantitatively confirmed by aligning its predictions with an expert-written baseline and qualitatively validated through manual soundness verification. We use AutoBaxBuilder to construct a new benchmark and release it to the public as AutoBaxBench, together with a thorough evaluation on contemporary LLMs. AutoBaxBuilder generates new tasks in under 2 hours, for less than USD 4. Including a manual verification, this reduces the required human effort for benchmark construction by a factor of 12.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
