What Makes a Good LLM Agent for Real-world Penetration Testing?
Nanyang Technological University(南洋理工大学);University of New South Wales(新南威尔士大学);Singapore Management University(新加坡管理大学);CFAR, A*STAR, Singapore(新加坡科技研究局计算未来与先进机器人中心);Tsinghua University(清华大学)
Gelei Deng, Yi Liu, Yuekang Li, Ruozhao Yang, Xiaofei Xie, Jie Zhang, Han Qiu, Tianwei Zhang
LLM-based agents show promise for automating penetration testing, yet reported performance varies widely across systems and benchmarks. We analyze 28 LLM-based penetration testing systems and evaluate five representative implementations across three benchmarks of increasing complexity. Our analysis reveals two distinct failure modes: Type A failures stem from capability gaps (missing tools, inadequate prompts) that engineering readily addresses, while Type B failures persist regardless of tooling due to planning and state management limitations. We show that Type B failures share a root cause that is largely invariant to the underlying LLM: agents lack real-time task difficulty estimation. As a result, agents misallocate effort, over-commit to low-value branches, and exhaust context before completing attack chains.
Based on this insight, we present Excalibur, a penetration testing agent that couples strong tooling with difficulty-aware planning. A Tool and Skill Layer eliminates Type A failures through typed interfaces and retrieval-augmented knowledge. A Task Difficulty Assessment (TDA) mechanism addresses Type B failures by estimating tractability through four measurable dimensions (horizon estimation, evidence confidence, context load, and historical success) and uses these estimates to guide exploration-exploitation decisions within an Evidence-Guided Attack Tree Search (EGATS) framework. Excalibur achieves up to 91% task completion on CTF benchmarks with frontier models (39 to 49% relative improvement over baselines) and compromises 4 of 5 hosts on the GOAD Active Directory environment versus 2 by prior systems. These results show that difficulty-aware planning yields consistent end-to-end gains across models and addresses a limitation that model scaling alone does not eliminate.
PentestEval: Benchmarking LLM-based Penetration Testing with Modular and Stage-Level Design
School of Computing and Information Systems, Singapore Management University(计算与信息系统学院,新加坡国立管理大学);School of Computer Science and Engineering, Nanyang Technological University(计算机科学与工程学院,南洋理工大学);School of Cyber Security, Tianjin University(网络安全学院,天津大学)
AutoPT: How Far Are We from the End2End Automated Web Penetration Testing?
University of Science and Technology of China(中国科学技术大学);QI-ANXIN Technology Research Institute(奇安信技术研究院);Chaitin Future Technology Co., Ltd(观想未来科技有限公司)
渗透测试是一种通过生成和执行可能的攻击来评估网络安全的方法。这样做可以自动进行定期和系统的测试,而无需大量的人力。一个关键问题是如何产生攻击。这自然是一个规划问题。以前的工作(Lucangeli et al. 2010)使用经典规划,因此忽略了所有黑客特有的不完整知识。最近的工作(Sarraute等人,2011)为了扩展而做出了强有力的独立性假设,并且缺乏关于攻击规划问题实际上是什么的明确的正式概念。在这里,我们模型的部分可观察马尔可夫决策过程(POMDP)的问题。这是渗透测试在一个良好的研究形式主义的基础,突出了这个问题的性质的重要方面。POMDPs允许将信息收集建模为问题的一个组成部分,从而首次提供了一种智能地将扫描操作与实际利用混合在一起的方法。
Modeling Penetration Testing with Reinforcement Learning Using Capture-the-Flag Challenges: Trade-offs between Model-free Learning and A Priori Knowledge
Guided Reasoning in LLM-Driven Penetration Testing Using Structured Attack Trees
Department of Electrical and Computer Engineering, Rochester Institute of Technology(罗切斯特理工学院电子与计算机工程系);Institute for Informatics and Applied Technology, Gonzaga University(贡萨加大学信息学与应用技术研究所)
Enhancing Security Testing Software for Systems that Cannot be Subjected to the Risks of Penetration Testing Through the Incorporation of Multi-threading and and Other Capabilities
Requirements and Recommendations for IoT/IIoT Models to automate Security Assurance through Threat Modelling, Security Analysis and Penetration Testing
Virtualization-based Penetration Testing Study for Detecting Accessibility Abuse Vulnerabilities in Banking Apps in East and Southeast Asia
Singapore Management University(新加坡管理大学);MIE-SPPU Institute of Higher Education(MIE-SPPU高等教育学院);i-Sprint Innovations Pte. Ltd(i-Sprint创新私人有限公司);Promon(普罗蒙)
Vulnerability Assessment and Penetration Testing on IP cameras
IP摄像机一直是物联网(IoT)的一部分,是家庭和专业环境中使用最广泛的设备之一。不幸的是,IP摄像机的漏洞吸引了恶意活动。例如,在2016年,一次大规模的攻击导致数千个摄像头和物联网设备被攻破,并被用来创建僵尸网络。考虑到这些设备的历史和所访问数据的极端敏感性,人们自然会质疑目前的安全措施。在本文中,漏洞评估和渗透测试进行了一个特定型号的IP摄像机,TP-Link Tapo C200。更详细地说,我们的调查结果表明,有问题的IP摄像机存在三个漏洞,例如:拒绝服务,视频窃听,最后是一种名为“Motion Oracle”的新型攻击。实验不仅限于攻击性部分,而且还为有问题的摄像机和所有可能遭受相同漏洞的人提出对策。对策是基于使用另一个物联网设备,Raspberry Pi。
Evaluation of Reinforcement Learning for Autonomous Penetration Testing using A3C, Q-learning and DQN
German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);OPS-GDA, ESA-ESOC(欧洲空间操作中心(ESA-ESOC)OPS-GDA部门);Technische Universität Kaiserslautern(凯泽斯劳滕工业大学)