多模态大语言模型中的安全几何坍缩与自适应漂移修正
Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correction
AI总结：
本文研究了多模态大语言模型在跨模态安全转移中的不足，提出安全几何坍缩现象，并通过自适应漂移修正方法提升模型安全性。
摘要：
多模态大语言模型（MLLMs）常常无法将文本模态中学习到的安全能力转移到语义等价的非文本输入中，揭示出一个持续存在的多模态安全缺口。我们从表示几何视角出发，通过分析文本对齐的拒绝方向和模态诱导的漂移方向来研究这一缺口。我们展示了多模态输入压缩了沿拒绝方向的可用分离度，使其不再可靠用于识别和拒绝有害输入。我们将这种失败模式称为安全几何坍缩。我们通过条件拒绝分离度量化这一现象，并显示更强的模态诱导漂移与更弱的拒绝分离度和更高的攻击成功率一致。随后，我们通过固定强度激活干预验证了模态诱导漂移的因果作用：抵消估计的漂移可以恢复拒绝分离度并提高多模态安全性。在漂移修正后，我们进一步观察到自修正现象，其中模型在前向动态中恢复了识别和拒绝有害多模态输入的能力。这种效果也提供了模型对每个输入感知有害性的内部信号。受此信号启发，我们提出了ReGap，一种无需训练的推理时方法，通过自修正自适应修正模态漂移。在多个多模态安全基准和实用性基准上的实验展示了ReGap的有效性，显著提高了MLLMs的安全性，而不会损害通用能力。我们的发现强调了表示层面的模态对齐作为实时安全改进和构建更安全、更可靠MLLMs的关键方向。
英文摘要：
Multimodal large language models (MLLMs) often fail to transfer safety capabilities learned in the text modality to semantically equivalent non-text inputs, revealing a persistent multimodal safety gap. We study this gap from a representation-geometric perspective by analyzing a text-aligned refusal direction and a modality-induced drift direction. We show that multimodal inputs compress the usable separation along the refusal direction, making it no longer reliable for identifying and refusing harmful inputs. We refer to this failure mode as Safety Geometry Collapse. We quantify it through conditional refusal separability and show that stronger modality-induced drift is consistently associated with weaker refusal separability and higher attack success rates. We then validate the causal role of modality-induced drift through a fixed-strength activation intervention: counteracting the estimated drift restores refusal separability and improves multimodal safety. After drift correction, we further observe self-rectification, where the model recovers its ability to recognize and refuse harmful multimodal inputs during forward dynamics. This effect also provides an internal signal of the model's perceived harmfulness of each input. Motivated by this signal, we propose ReGap, a training-free inference-time method that adaptively corrects modality drift using self-rectification. Experiments across multiple multimodal safety benchmarks and utility benchmarks demonstrate the effectiveness of ReGap, which significantly improves the safety of MLLMs without compromising general capabilities. Our findings highlight representation-level modality alignment as a crucial direction for real-time safety improvement and for building safer, more reliable MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
