跨模态冲突下的全方位安全：漏洞、动态机制和高效对齐
Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignment
AI总结：
本文提出OmniSteer方法，通过提取黄金拒绝向量和轻量级适配器提升多模态模型的安全性与通用能力。
摘要：
全方位大语言模型（OLLMs）大大扩展了LLM的多模态能力，但也引入了跨模态安全风险。然而，对全方位多模态交互中漏洞的系统理解仍然缺乏。为弥合这一差距，我们建立了模态-语义解耦原则，并构建了AdvBench-Omni数据集，揭示了OLLMs中显著的漏洞。机理分析揭示了由拒绝向量幅度缩小驱动的中层溶解现象，以及存在模态不变的纯拒绝方向。受这些见解启发，我们利用奇异值分解提取出黄金拒绝向量，并提出OmniSteer，该方法利用轻量级适配器来适应性地调节干预强度。广泛的实验表明，我们的方法不仅将对有害输入的拒绝成功率从69.9%提高到91.2%，还能在所有模态中有效保持通用能力。我们的代码可在：https://github.com/zhrli324/omni-safety-research 上获得。
英文摘要：
Omni-modal Large Language Models (OLLMs) greatly expand LLMs' multimodal capabilities but also introduce cross-modal safety risks. However, a systematic understanding of vulnerabilities in omni-modal interactions remains lacking. To bridge this gap, we establish a modality-semantics decoupling principle and construct the AdvBench-Omni dataset, which reveals a significant vulnerability in OLLMs. Mechanistic analysis uncovers a Mid-layer Dissolution phenomenon driven by refusal vector magnitude shrinkage, alongside the existence of a modal-invariant pure refusal direction. Inspired by these insights, we extract a golden refusal vector using Singular Value Decomposition and propose OmniSteer, which utilizes lightweight adapters to modulate intervention intensity adaptively. Extensive experiments show that our method not only increases the Refusal Success Rate against harmful inputs from 69.9% to 91.2%, but also effectively preserves the general capabilities across all modalities. Our code is available at: https://github.com/zhrli324/omni-safety-research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
