arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.25705cs.CRcs.SE
收藏

安全技能在Linux内核漏洞检测中对编码智能体的指导效果如何?

How Effectively Do Security Skills Guide Coding Agents in Vulnerability Detection in Linux Kernel?

York University(约克大学)

Haoran Xue, Nikta Akbarpour, Gias Uddin, Song Wang

AI总结:

本研究通过大规模实证研究分析安全技能对编码智能体的漏洞检测效果,提出SKILL-SIF自改进框架,可显著提升Linux内核漏洞检测准确率。

摘要:

大型语言模型(LLM)编码智能体(如Claude Code)正越来越多地用于安全审计,特定任务技能提供可复用的漏洞分析流程。然而,它们检测真实漏洞的效果及失败原因仍不明确。我们开展了首个大规模实证研究:使用Claude Code搭配4种底层LLM,针对10类共100个真实Linux内核漏洞,测试37种安全技能,总计完成14800次运行。我们在函数、语句、漏洞类别三个层面评估漏洞检测效果。结果显示,24.4%的运行定位到了正确函数,18.3%同时识别出函数及负责语句,15.7%正确分类了漏洞。失败分析揭示了其在证据获取、状态关系推理、代码解释、候选验证、调查覆盖范围及报告方面的局限性。基于这些发现,我们提出SKILL-SIF,这是一种自改进框架,通过从遗漏漏洞和失败模式中学习,迭代优化安全技能。我们在原始基准上优化了5种表现最佳的技能,并在100个未见过的漏洞上进行评估。SKILL-SIF使未见过漏洞的完全正确检测率提升了最多8倍(从2%升至16%),同时4种LLM的综合准确率提升了1.85倍(从6.05%升至11.20%)。这些发现凸显了现有安全技能的局限性,也证明了失败驱动的技能优化可提升编码智能体的漏洞检测能力。

英文摘要:

Large language model (LLM) coding agents, such as Claude Code, are increasingly used for security auditing, with task-specific skills providing reusable vulnerability-analysis procedures. However, their effectiveness in detecting real vulnerabilities and the reasons for their failures remain unclear. We conduct the first large-scale empirical study of 37 security skills using Claude Code with four underlying LLMs on 100 real Linux kernel vulnerabilities across ten classes, totaling 14800 runs. We evaluate vulnerability detection at the function, statement, and vulnerability-class levels. Results show that 24.4 percent of runs locate the correct function, 18.3 percent identify both the function and responsible statement, and 15.7 percent correctly classify the vulnerability. Our failure analysis reveals limitations in evidence acquisition, state-relation reasoning, code interpretation, candidate verification, investigation coverage, and reporting. Based on these findings, we propose SKILL-SIF, a self-improving framework that iteratively evolves security skills by learning from missed vulnerabilities and failure patterns. We evolve five top-performing skills using the original benchmark and evaluate them on 100 previously unseen vulnerabilities. SKILL-SIF improves fully correct vulnerability detection by up to eight times (from 2 percent to 16 percent) on unseen vulnerabilities, while aggregate accuracy across four LLMs increases by 1.85 times (from 6.05 percent to 11.20 percent). These findings highlight the limitations of existing security skills and demonstrate the potential of failure-driven skill evolution to improve vulnerability detection in coding agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑