arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.23633cs.SE
收藏

检测-修复-验证LLM生成的代码:多语言、多粒度实证研究

Detect--Repair--Verify for LLM-Generated Code: A Multi-Language, Multi-Granularity Empirical Study

Concordia University(麦科德大学)

Cheng Cheng

AI总结:

本文通过检测-修复-验证工作流评估LLM生成代码的安全性,发现多阶段修复能提升安全性和正确性,但修复可靠性依赖于修复范围和测试基础。

摘要:

大型语言模型可以生成可运行的软件构件,但其安全性难以端到端评估。本研究通过检测-修复-验证(DRV)工作流探讨该问题,其中漏洞被检测、修复并重新检查安全性和功能性测试。研究填补了四个空白:缺乏基于测试的LLM生成构件基准、管道级效果证据不足、检测报告的可靠性不明确、验证下修复的可信度不确定。为支持本研究,构建了EduCollab作为多语言、多粒度的可运行LLM生成Web应用基准,涵盖PHP、JavaScript和Python。每个构件配以可执行的功能和漏洞测试套件,基准涵盖项目、需求和文件级别设置。在该基准上,研究比较了未修复基线、单次检测-修复和有限迭代DRV在可比预算下的表现。结果通过安全且正确的产出衡量,并分析中间构件和迭代轨迹以评估报告的可行性和修复失败模式。结果表明,有限迭代DRV能提升安全且正确的产出,但项目层面的收益不均,修复范围更窄时收益更明显。检测报告通常对后续修复有用,但其可靠性不一致。修复可信度也强烈依赖于修复范围。这些发现突显了需要基于测试的端到端评估LLM基于漏洞管理流程的必要性。

英文摘要:

Large language models can generate runnable software artifacts, but their security remains difficult to evaluate end to end. This study examines that problem through a Detect--Repair--Verify (DRV) workflow, in which vulnerabilities are detected, repaired, and then rechecked with security and functional tests. It addresses four gaps in current evidence: the lack of test-grounded benchmarks for LLM-generated artifacts, limited evidence on pipeline-level effectiveness, unclear reliability of detection reports as repair guidance, and uncertain repair trustworthiness under verification. To support this study, EduCollab is constructed as a multi-language, multi-granularity benchmark of runnable LLM-generated web applications in PHP, JavaScript, and Python. Each artifact is paired with executable functional and exploit test suites, and the benchmark spans project-, requirement-, and file-level settings. On this benchmark, the study compares unrepaired baselines, single-pass detect--repair, and bounded iterative DRV under comparable budget constraints. Outcomes are measured by secure-and-correct yield, and intermediate artifacts and iteration traces are analyzed to assess report actionability and repair failure modes. The results show that bounded iterative DRV can improve secure-and-correct yield over single-pass repair, but the gains are uneven at the project level and become clearer at narrower repair scopes. Detection reports are often useful for downstream repair, but their reliability is inconsistent. Repair trustworthiness also depends strongly on repair scope. These findings highlight the need for test-grounded, end-to-end evaluation of LLM-based vulnerability management workflows.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑