arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.20945cs.CRcs.LG
收藏

打破坏:基于可解释性的最新LLM安全审计

Breaking Bad: Interpretability-Based Safety Audits of State-of-the-Art LLMs

NuSCI Research Group, Computer Science Laboratory, SRI(NuSCI研究组、计算机科学实验室、SRI)

Krishiv Agarwal, Ramneet Kaur, Colin Samplawski, Manoj Acharya, Anirban Roy, Daniel Elenius, Brian Matejek, Adam D. Cobb, Susmit Jha

AI总结:

本文提出基于可解释性的 jailbreaking 审计方法,评估了八种最新开源LLM的安全性,发现Llama-3模型易受攻击,而GPT-oss-120B表现稳健,Qwen和Phi模型结果混杂,揭示了可解释性工具在安全审计中的有效性及潜在风险。

摘要:

有效的大型语言模型(LLM)安全审计需要超越黑箱探测的工具,系统性地揭示模型内部的漏洞。我们提出了一种全面的、基于可解释性的jailbreaking审计,评估了八种最先进的开源LLM:Llama-3.1-8B、Llama-3.3-70B-4bt、GPT-oss-20B、GPT-oss-120B、Qwen3-0.6B、Qwen3-32B、Phi4-3.8B和Phi4-14B。利用基于可解释性的方法——通用引导(US)和表示工程(RepE)——我们引入了一种自适应的两阶段网格搜索算法,以确定最优的激活引导系数,用于不安全行为概念。我们的评估在经过筛选的有害查询集和标准化的LLM判断协议上进行,揭示了模型鲁棒性的显著差异。Llama-3模型高度易受攻击,Llama-3.3-70B-4bt上的jailbroken响应高达91%(US)和83%(RepE),而GPT-oss-120B在两种可解释性方法下均保持稳健。Qwen和Phi模型的结果混合,较小的Qwen3-0.6B和Phi4-3.8B大多表现出较低的jailbreaking率,而它们的较大版本则更易受攻击。我们的结果证明了基于可解释性的引导作为系统安全审计的强大工具,但也突显了其双用途风险以及在LLM部署中需要更好的内部防御的必要性。

英文摘要:

Effective safety auditing of large language models (LLMs) demands tools that go beyond black-box probing and systematically uncover vulnerabilities rooted in model internals. We present a comprehensive, interpretability-driven jailbreaking audit of eight SOTA open-source LLMs: Llama-3.1-8B, Llama-3.3-70B-4bt, GPT-oss- 20B, GPT-oss-120B, Qwen3-0.6B, Qwen3-32B, Phi4-3.8B, and Phi4-14B. Leveraging interpretability-based approaches -- Universal Steering (US) and Representation Engineering (RepE) -- we introduce an adaptive two-stage grid search algorithm to identify optimal activation-steering coefficients for unsafe behavioral concepts. Our evaluation, conducted on a curated set of harmful queries and a standardized LLM-based judging protocol, reveals stark contrasts in model robustness. The Llama-3 models are highly vulnerable, with up to 91\% (US) and 83\% (RepE) jailbroken responses on Llama-3.3-70B-4bt, while GPT-oss-120B remains robust to attacks via both interpretability approaches. Qwen and Phi models show mixed results, with the smaller Qwen3-0.6B and Phi4-3.8B mostly exhibiting lower jailbreaking rates, while their larger counterparts are more susceptible. Our results establish interpretability-based steering as a powerful tool for systematic safety audits, but also highlight its dual-use risks and the need for better internal defenses in LLM deployment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑