arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.16931cs.AI
收藏

细粒度微调会削弱视觉语言代理的安全对齐

Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agents

Idhant Gulati, Shivam Raval

AI总结:

研究发现细粒度微调在有害数据集上会导致广泛的任务和模态对齐偏差,且单模态安全基准可能低估视觉语言模型的对齐退化。

摘要:

终身多模态代理必须通过训练后持续适应新任务,但这就产生了能力获取与安全对齐保护之间的根本矛盾。我们证明了在狭窄领域有害数据集上微调对齐的视觉语言模型会引发严重的新兴偏差,这种偏差能广泛泛化到无关任务和模态。通过在Gemma3-4B上的实验,我们显示偏差随LoRA秩单调增加,且多模态评估显示偏差(r=128时70.71±1.22)比单模态评估(41.19±2.51)高得多,表明单模态安全基准可能低估视觉语言模型的对齐退化。关键发现是即使训练混合中有10%的有害数据也会导致显著的对齐退化。几何分析显示有害行为占据了一个显著低维子空间,大部分偏差信息被捕捉在10个主成分中。为缓解偏差,我们评估了两种策略:良性细粒度微调和基于激活的引导。虽然这两种方法显著减少了偏差,但并未完全消除学习到的有害行为。我们的发现强调了需要鲁棒的持续学习框架,因为当前训练后范式可能无法在部署后充分保持对齐。

英文摘要:

Lifelong multimodal agents must continuously adapt to new tasks through post-training, but this creates a fundamental tension between acquiring capabilities and preserving safety alignment. We demonstrate that fine-tuning aligned vision-language models on narrow-domain harmful datasets induces severe emergent misalignment that generalizes broadly across unrelated tasks and modalities. Through experiments on Gemma3-4B, we show that misalignment scales monotonically with LoRA rank, and that multimodal evaluation reveals substantially higher misalignment ($70.71 \pm 1.22$ at $r=128$) than text-only evaluation ($41.19 \pm 2.51$), suggesting that unimodal safety benchmarks may underestimate alignment degradation in vision-language models. Critically, even 10\% harmful data in the training mixture induces substantial alignment degradation. Geometric analysis reveals that harmful behaviors occupy a remarkably low-dimensional subspace, with the majority of misalignment information captured in 10 principal components. To mitigate misalignment, we evaluate two strategies: benign narrow fine-tuning and activation-based steering. While both approaches substantially reduce misalignment, neither completely removes the learned harmful behaviors. Our findings highlight the need for robust continual learning frameworks, as current post-training paradigms may not sufficiently preserve alignment in post-deployment settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑