arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.17622cs.CRcs.SE
收藏

什么样的LLM代理适合现实中的渗透测试?

What Makes a Good LLM Agent for Real-world Penetration Testing?

Nanyang Technological University(南洋理工大学);University of New South Wales(新南威尔士大学);Singapore Management University(新加坡管理大学);CFAR, A*STAR, Singapore(新加坡科技研究局计算未来与先进机器人中心);Tsinghua University(清华大学)

Gelei Deng, Yi Liu, Yuekang Li, Ruozhao Yang, Xiaofei Xie, Jie Zhang, Han Qiu, Tianwei Zhang

AI总结:

Excalibur通过结合强大工具和难度感知计划,提高了渗透测试代理在复杂任务中的表现和效率。

摘要:

基于LLM的代理在自动化渗透测试中显示出潜力,但不同系统和基准测试的报告性能差异很大。我们分析了28个基于LLM的渗透测试系统,并在三个复杂度逐渐增加的基准测试中评估了五个代表性实现。我们的分析揭示了两种不同的失败模式:类型A的失败源于能力缺口(缺少工具、提示不足),这通过工程手段可以轻易解决;而类型B的失败则无论工具如何都存在,原因是计划和状态管理的限制。我们表明,类型B的失败有一个根本原因,基本上不受底层LLM的影响:代理缺乏实时任务难度估计。因此,代理会错误分配努力,过度承诺低价值分支,并在完成攻击链之前耗尽上下文。基于这一见解,我们提出了Excalibur,一种结合强大工具和难度感知计划的渗透测试代理。一个工具和技能层通过类型化的接口和检索增强的知识消除了类型A的失败。一个任务难度评估(TDA)机制通过四个可测量的维度(时间估计、证据信心、上下文负荷和历史成功)估计可操作性,并利用这些估计来指导探索-利用决策,以在证据引导的攻击树搜索(EGATS)框架中进行。Excalibur在CTF基准测试中实现了高达91%的任务完成率,使用前沿模型(相对于基线有39至49%的相对提升),并在GOAD活动目录环境中占有了4个中的5个主机,而先前的系统只占2个。这些结果表明,难度感知的计划在模型和模型扩展方面都带来了持续的端到端收益,并解决了模型扩展本身无法消除的限制。

英文摘要:

LLM-based agents show promise for automating penetration testing, yet reported performance varies widely across systems and benchmarks. We analyze 28 LLM-based penetration testing systems and evaluate five representative implementations across three benchmarks of increasing complexity. Our analysis reveals two distinct failure modes: Type A failures stem from capability gaps (missing tools, inadequate prompts) that engineering readily addresses, while Type B failures persist regardless of tooling due to planning and state management limitations. We show that Type B failures share a root cause that is largely invariant to the underlying LLM: agents lack real-time task difficulty estimation. As a result, agents misallocate effort, over-commit to low-value branches, and exhaust context before completing attack chains. Based on this insight, we present Excalibur, a penetration testing agent that couples strong tooling with difficulty-aware planning. A Tool and Skill Layer eliminates Type A failures through typed interfaces and retrieval-augmented knowledge. A Task Difficulty Assessment (TDA) mechanism addresses Type B failures by estimating tractability through four measurable dimensions (horizon estimation, evidence confidence, context load, and historical success) and uses these estimates to guide exploration-exploitation decisions within an Evidence-Guided Attack Tree Search (EGATS) framework. Excalibur achieves up to 91% task completion on CTF benchmarks with frontier models (39 to 49% relative improvement over baselines) and compromises 4 of 5 hosts on the GOAD Active Directory environment versus 2 by prior systems. These results show that difficulty-aware planning yields consistent end-to-end gains across models and addresses a limitation that model scaling alone does not eliminate.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑