理解与纠正视觉语言模型（VLMs）中的安全感知扭曲
Understanding and Rectifying Safety Perception Distortion in VLMs
AI总结：
针对VLMs集成视觉模态后更易受有害请求攻击的问题，本文提出无需训练的激活偏移解耦与校准（ShiftDC）方法，通过分解校准模态诱导的激活偏移，在保留模型能力的同时恢复其安全对齐性，显著提升安全基准性能。
摘要：
近期研究表明，视觉语言模型（VLMs）在集成视觉模态后，比其纯文本大型语言模型（LLM）骨干更易受到有害请求和越狱攻击，表现出更大的脆弱性。为揭示该现象的根本原因，我们开展深入分析并发现一个关键问题：与纯文本对应模型相比，多模态输入会引发模态诱导的激活向“更安全”方向偏移，导致VLMs系统性高估有害输入的安全性。我们将此问题称为安全感知扭曲。为缓解这种扭曲，我们提出激活偏移解耦与校准（ShiftDC）方法——一种无需训练的方法，通过分解和校准模态诱导的激活偏移来降低模态对安全性的影响。ShiftDC通过分离并移除安全相关组件，在保留VLMs视觉语言能力的同时，恢复LLM骨干固有的安全对齐性。实证结果表明，ShiftDC在安全基准上显著提升对齐性能，且不损害模型效用。
英文摘要：
Recent studies reveal that vision-language models (VLMs) become more susceptible to harmful requests and jailbreak attacks after integrating the vision modality, exhibiting greater vulnerability than their text-only LLM backbones. To uncover the root cause of this phenomenon, we conduct an in-depth analysis and identify a key issue: multimodal inputs introduce an modality-induced activation shift toward a "safer" direction compared to their text-only counterparts, leading VLMs to systematically overestimate the safety of harmful inputs. We refer to this issue as safety perception distortion. To mitigate such distortion, we propose Activation Shift Disentanglement and Calibration (ShiftDC), a training-free method that decomposes and calibrates the modality-induced activation shift to reduce the impact of modality on safety. By isolating and removing the safety-relevant component, ShiftDC restores the inherent safety alignment of the LLM backbone while preserving the vision-language capabilities of VLMs. Empirical results demonstrate that ShiftDC significantly enhances alignment performance on safety benchmarks without impairing model utility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
