arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.02873cs.SEcs.AIcs.CR
收藏

大语言模型安全工具编排的决定因素与限制:基于HexStrike-AI的研究

Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AI

University of Science and Technology of China(中国科学技术大学);Suzhou Institute for Advanced Research(苏州研究院)

Romain Gerard, Assmaa Zeghaider, Yan Guo

AI总结:

以HexStrikeAI为测试平台,通过评估系统、诊断故障并改进,运行多类挑战,发现驱动客户端是固定模型的首要因素,整体解决率提升,剩余故障受推理或环境限制,还讨论了编排器评估及局限性。

摘要:

通过模型上下文协议驱动安全工具套件的大语言模型代理越来越普遍。然而,限制其能力的因素仍未得到充分描述。我们以开源编排器HexStrikeAI为测试平台,采用评估系统、诊断故障并进行针对性改进的方法。运行多类挑战并改进后,解决率提升,剩余故障受推理或环境限制。还讨论了结果对编排器评估的意义及局限性。

英文摘要:

Large language model agents driving security tool suites over the Model Context Protocol are increasingly common. Yet the factors that bound their capability remain poorly characterized: how much depends on the model versus the client that drives it, whether constraining the agent to the orchestrator's own tools helps, and where capability is limited by reasoning rather than by missing tools. Using HexStrikeAI, an open-source orchestrator that exposes 150+ tools, as a testbed, we follow a methodology that evaluates the system, diagnoses its failures, and applies targeted improvements. We run 86 picoCTF challenges across seven categories and three difficulty tiers, under three tool-access regimes and three model/client configurations (774 trials). We then apply corrections to existing tools, agent-behavior changes, and eleven new capability tools, and re-run the previously-unsuccessful trials. The diagnosis isolates the driving client as a first-order factor for a fixed model (a 2.1 * gap between two DeepSeek clients) and a monotonic difficulty gradient, with the largest gains in the mid tier. The overall solve rate rises from 55.4% to 72.0%, and every configuration improves significantly (paired McNemar p < 0.001, non-overlapping 95% confidence intervals). The residual failures are reasoning- or environment-bound rather than missing-tool. A 60-run stability sub-study finds single-run verdicts reproducible (17/20 unanimous). We discuss what the results imply for how such orchestrators should be evaluated, and we are explicit about the limits: the study uses a single benchmark, the fixes were tuned on the same challenges they were evaluated on, and the client effect is demonstrated for one model only, so its generality to other models remains a hypothesis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑