RealSec-bench：一个评估现实世界仓库中安全代码生成的基准
RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories
AI总结：
RealSec-bench是一个基于现实世界Java仓库构建的安全代码生成评估基准，通过多阶段流程和专家验证，评估5种LLM在功能正确性和安全性方面的表现。
摘要：
大型语言模型（LLMs）在代码生成方面展示了显著的能力，但它们在生成安全代码方面的熟练程度仍然是一个关键但未充分研究的领域。现有基准测试往往不足，因为它们依赖于合成漏洞或孤立评估功能正确性，无法捕捉到现实世界软件中功能与安全之间复杂的相互作用。为了解决这一差距，我们引入了RealSec-bench，一个新的安全代码生成基准测试，其精心构建于现实世界、高风险的Java仓库中。我们的方法采用多阶段流程，结合系统性的SAST扫描与CodeQL、基于LLM的假阳性消除以及严格的专家验证。所得到的基准测试包含105个基于现实仓库情境的实例，涵盖19种常见的弱点枚举（CWE）类型，并表现出广泛的数据流复杂性，包括具有多达34跳过程依赖性的漏洞。使用RealSec-bench，我们对5种流行的LLM进行了广泛的实证研究。我们引入了一个新的复合指标SecurePass@K，以同时评估功能正确性和安全性。我们发现，尽管检索增强生成（RAG）技术可以提高功能正确性，但它们对安全性提供 negligible 的好处。此外，显式提示模型使用通用安全指南往往会导致编译失败，损害功能正确性，而无法可靠地防止漏洞。我们的工作突显了当前LLM在功能和安全代码生成之间的差距。
英文摘要：
Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area. Existing benchmarks often fall short by relying on synthetic vulnerabilities or evaluating functional correctness in isolation, failing to capture the complex interplay between functionality and security found in real-world software. To address this gap, we introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories. Our methodology employs a multi-stage pipeline that combines systematic SAST scanning with CodeQL, LLM-based false positive elimination, and rigorous human expert validation. The resulting benchmark contains 105 instances grounded in real-word repository contexts, spanning 19 Common Weakness Enumeration (CWE) types and exhibiting a wide diversity of data flow complexities, including vulnerabilities with up to 34-hop inter-procedural dependencies. Using RealSec-bench, we conduct an extensive empirical study on 5 popular LLMs. We introduce a novel composite metric, SecurePass@K, to assess both functional correctness and security simultaneously. We find that while Retrieval-Augmented Generation (RAG) techniques can improve functional correctness, they provide negligible benefits to security. Furthermore, explicitly prompting models with general security guidelines often leads to compilation failures, harming functional correctness without reliably preventing vulnerabilities. Our work highlights the gap between functional and secure code generation in current LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
