arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.07830cs.CRcs.AI
收藏

CyBiasBench:用于网络攻击场景的LLM代理偏见基准测试

CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios

Chung-Ang University(Chung-Ang大学);Myongji University(Myongji大学)

Taein Lim, Seongyong Ju, Munhyeok Kim, Hyunjun Kim, Hoki Kim

AI总结:

本文提出CyBiasBench,通过630次会话评估五个代理在三种目标和四种提示条件下的攻击偏见,揭示代理在攻击家族分配上的差异及偏见动量效应。

摘要:

本文揭示了大型语言模型(LLMs)作为自主代理在进攻性网络安全中的一个有趣现象:不同代理表现出不同的攻击模式。具体而言,每个代理都表现出攻击选择偏差,无论提示变化如何,都会倾向于集中在一小部分攻击家族上。为系统量化这种行为,我们引入了CyBiasBench,一个涵盖630次会话的综合性基准测试,评估五个代理在三种目标和四种提示条件下的十种攻击家族。我们发现代理之间存在明显的偏见,不同代理主导的攻击家族和攻击家族分配分布的熵水平各不相同。这种偏见更好地被描述为代理的特性,而不是与攻击成功率相关的因素。此外,我们的实验揭示了偏见动量效应,即代理会抵制明确转向与自身偏见冲突的攻击家族。这种强制分布转移并未带来可测量的攻击性能提升。为确保可重复性和促进未来研究,我们发布了交互式结果仪表板,并提供了可重复性存档,包含聚合的会话级统计数据和完整的评估脚本。

英文摘要:

Large language models (LLMs) are increasingly deployed as autonomous agents in offensive cybersecurity. In this paper, we reveal an interesting phenomenon: different agents exhibit distinct attack patterns. Specifically, agents exhibit distinct attack-selection biases, differing in both the families they explore and the concentration of their efforts. These patterns remain relatively stable across prompt variations. To systematically quantify this behavior, we introduce CyBiasBench, a comprehensive 630-session benchmark that evaluates five agents on three targets and four prompt conditions with ten attack families. We identify explicit bias across agents, with different dominant attack families and varying entropy levels in their attack-family allocation distributions. Such bias characterizes agent behavior within the evaluated environments and provides information beyond aggregate attack success rates. Furthermore, our experiments reveal a bias momentum effect, where agents resist explicit steering toward attack families that conflict with their bias. This forced distribution shift does not yield measurable improvements in attack performance. To ensure reproducibility and facilitate future research, we release an interactive result dashboard at https://trustworthyai.co.kr/CyBiasBench/ and a reproducibility artifact with aggregated session-level statistics and full evaluation scripts at https://github.com/Harry24k/CyBiasBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑