Large language model agents driving security tool suites over the Model Context Protocol are increasingly common. Yet the factors that bound their capability remain poorly characterized: how much depends on the model versus the client that drives it, whether constraining the agent to the orchestrator's own tools helps, and where capability is limited by reasoning rather than by missing tools. Using HexStrikeAI, an open-source orchestrator that exposes 150+ tools, as a testbed, we follow a methodology that evaluates the system, diagnoses its failures, and applies targeted improvements. We run 86 picoCTF challenges across seven categories and three difficulty tiers, under three tool-access regimes and three model/client configurations (774 trials). We then apply corrections to existing tools, agent-behavior changes, and eleven new capability tools, and re-run the previously-unsuccessful trials. The diagnosis isolates the driving client as a first-order factor for a fixed model (a 2.1 * gap between two DeepSeek clients) and a monotonic difficulty gradient, with the largest gains in the mid tier. The overall solve rate rises from 55.4% to 72.0%, and every configuration improves significantly (paired McNemar p < 0.001, non-overlapping 95% confidence intervals). The residual failures are reasoning- or environment-bound rather than missing-tool. A 60-run stability sub-study finds single-run verdicts reproducible (17/20 unanimous). We discuss what the results imply for how such orchestrators should be evaluated, and we are explicit about the limits: the study uses a single benchmark, the fixes were tuned on the same challenges they were evaluated on, and the client effect is demonstrated for one model only, so its generality to other models remains a hypothesis.
Advancing LLM-Based Security Automation with Customized Group Relative Policy Optimization for Zero-Touch Networks
National Engineering Research Center for Mobile Network Technologies, Beijing University of Posts and Telecommunications(移动网络技术国家工程研究中心,北京邮电大学);Beijing University of Posts and Telecommunications(北京邮电大学);Singapore University of Technology and Design(新加坡科技设计大学);Department of Electronic Engineering, Kyung Hee University(韩国庆熙大学电子工程系)
Agentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration Testing
Ahsanullah University of Science and Technology(阿赫桑乌拉科技大学);BRAC University(BRAC大学);North South University(北南大学);Missouri State University(密苏里州立大学);Multimedia University(多媒体大学);American International University-Bangladesh(孟加拉国美国国际大学)
DURA-CPS: A Multi-Role Orchestrator for Dependability Assurance in LLM-Enabled Cyber-Physical Systems
Cyrion Labs(Cyrion实验室);Carnegie Mellon University(卡内基梅隆大学);Cornell University(康奈尔大学);University of California San Diego(加州大学圣地亚哥分校);University of Montana(蒙大拿大学);Oakland University(奥克兰大学);University of Pennsylvania(宾夕法尼亚大学)