arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.05523cs.SEcs.AI
收藏

捕获旗帜:通过语义保持变换进行基于家庭的代理LLM评估

Capture the Flags: Family-Based Evaluation of Agentic LLMs via Semantics-Preserving Transformations

Department of Computing, Imperial College London(帝国理工学院伦敦计算机系);AI Security Institute(人工智能安全研究所);Royal Grammar School Guildford(格里菲斯皇家文法学校)

Shahin Honarvar, Amber Gorzynski, James Lee-Jones, Harry Coppock, Marek Rei, Joseph Ryan, Alastair F. Donaldson

AI总结:

本文提出CTF挑战家族,通过语义保持的程序变换生成等效挑战,评估代理鲁棒性和泛化能力,展示了Evolve-CTF工具及13种代理LLM配置的评估结果。

摘要:

代理大语言模型(LLM)越来越多地通过捕获旗帜(CTF)基准进行网络安全任务评估,但现有点评估基准对代理鲁棒性和跨源代码变种的泛化能力提供有限见解。我们引入CTF挑战家族,通过语义保持的程序变换生成等效挑战,从而在保持底层利用策略不变的情况下进行受控评估。我们提出了Evolve-CTF工具,该工具通过多种变换从Python挑战生成CTF家族。使用Evolve-CTF从Cybench和Intercode挑战推导家族,我们评估了13种代理LLM配置。我们发现模型对重命名和代码插入具有显著鲁棒性,但组合变换和更深层次的混淆会降低性能,要求更复杂的工具使用。显式推理对成功率影响有限。我们的工作贡献了一种技术及工具用于未来的LLM评估,并提供了一个大规模数据集,表征了当前最先进模型在此领域的能力。

英文摘要:

Agentic large language models (LLMs) are increasingly evaluated on cybersecurity tasks using capture-the-flag (CTF) benchmarks, yet existing pointwise benchmarks offer limited insight into agent robustness and generalisation across alternative versions of the source code. We introduce CTF challenge families, whereby a single CTF is used to generate a family of semantically-equivalent challenges via semantics-preserving program transformations, enabling controlled evaluation of robustness while keeping the underlying exploit strategy fixed. We present Evolve-CTF, a tool that generates CTF families from Python challenges using a range of transformations. Using Evolve-CTF to derive families from Cybench and Intercode challenges, we evaluate 13 agentic LLM configurations with tool access. We find that models are remarkably robust to renaming and code insertion, but that composed transformations and deeper obfuscation degrade performance by requiring more sophisticated tool use. Enabling explicit reasoning has little effect on success rates. Our work contributes a technique and tool for future LLM evaluations, and a large dataset characterising the capabilities of current state-of-the-art models in this domain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑