A Survey of LLM-Driven Penetration Testing: Taxonomy, Co-Evolution, and Open Challenges
University of Electronic Science and Technology of China(电子科技大学);The Hong Kong Polytechnic University(香港理工大学);Nanyang Technological University(南洋理工大学);Four-dimensional Powerise Technology Development Co., Ltd(四维力思科技发展有限公司);Jiangxi Academy of Cyber Security(江西省网络安全研究院)
Zheyuan He, Jiaxun Dong, Zihao Li, Ting Chen, Gelei Deng, Feng Luo, Jinkun Ji, Yuanlong Cao, Xiapu Luo
Agents4Pentest, an emerging class of LLM-based autonomous penetration testing systems, has become a rapidly growing area in security research. Despite this growth, the field still lacks a unified taxonomy, a systematic understanding of how agent architectures and evaluation benchmarks have co-evolved, and a clear characterization of remaining capability and reliability gaps. This survey addresses these gaps through a systematic analysis of 81 papers between 2023 and 2026. We organize the literature into six categories: evaluation benchmarks, general-purpose systems, domain-specific frameworks, CTF-based systems, defense-oriented research, and surveys. We further trace a four-phase architectural evolution from text-only reasoning agents to agents trained with Reinforcement Learning with Verifiable Rewards (RLVR), showing that each transition is driven by a distinct capability bottleneck. Our analysis yields several key findings. First, RLVR marks a shift in capability acquisition from imitation of expert demonstrations to reward-driven self-improvement, enabling agents to discover previously undocumented attack strategies. Second, CTF platforms have evolved from evaluation testbeds into dual-purpose infrastructure for both agent evaluation and RL training. Third, domain-specific frameworks improve efficiency through recurring specialization mechanisms, but their gains remain largely confined to narrow task classes and are difficult to compare across domains because existing evaluations rely on different benchmarks. Fourth, the field is expanding beyond offensive automation toward adversarial defense and security compliance. Across these categories, we identify three structurally linked open challenges: evaluation reliability, limited performance on multi-stage attack scenarios, and scarcity of high-quality training data.
Navigating the growing field of research on AI for software testing -- the taxonomy for AI-augmented software testing and an ontology-driven literature survey
PentestEval: Benchmarking LLM-based Penetration Testing with Modular and Stage-Level Design
School of Computing and Information Systems, Singapore Management University(计算与信息系统学院,新加坡国立管理大学);School of Computer Science and Engineering, Nanyang Technological University(计算机科学与工程学院,南洋理工大学);School of Cyber Security, Tianjin University(网络安全学院,天津大学)
What Makes a Good LLM Agent for Real-world Penetration Testing?
Nanyang Technological University(南洋理工大学);University of New South Wales(新南威尔士大学);Singapore Management University(新加坡管理大学);CFAR, A*STAR, Singapore(新加坡科技研究局计算未来与先进机器人中心);Tsinghua University(清华大学)
Virtualization-based Penetration Testing Study for Detecting Accessibility Abuse Vulnerabilities in Banking Apps in East and Southeast Asia
Singapore Management University(新加坡管理大学);MIE-SPPU Institute of Higher Education(MIE-SPPU高等教育学院);i-Sprint Innovations Pte. Ltd(i-Sprint创新私人有限公司);Promon(普罗蒙)
Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing
School of Cyber Science and Engineering, Sichuan University(四川大学网络空间安全学院);Institute for Network Sciences and Cyberspace, Tsinghua University(清华大学网络科学与网络空间研究院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);National University of Singapore(新加坡国立大学);College of Electronic Engineering, National University of Defense Technology(国防科技大学电子工程学院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院)
What Bugs Do Prolog Students Write? An Empirical Taxonomy and Data-Driven Mutation Framework
INESC-ID / IST, Universidade de Lisboa(里斯本大学);Artificial Intelligence Research Institute, Consejo Superior de Investigaciones Científicas(西班牙国家研究委员会人工智能研究所);Carnegie Mellon University(卡内基梅隆大学)
School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院);Jisuan Institute of Technology, Beijing JudaoYouda Network Technology Co. Ltd.(智算技术研究所,北京judaoYouda网络技术有限公司)