arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2603.19092cs.CVcs.AIcs.CLcs.LG

SAVeS: 通过语义线索引导视觉-语言模型中的安全判断

SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues

King Abdullah University of Science and Technology(国王阿卜杜勒-阿齐兹大学科学与技术学院);Technical University of Munich(慕尼黑技术大学)

Carlos Hinojosa, Clemens Grange, Bernard Ghanem

AI总结:

研究通过语义线索引导视觉-语言模型的安全判断,提出SAVeS基准和评估协议,验证安全决策对语义线索的敏感性,揭示模型对视觉-语言关联的依赖及潜在安全漏洞。

摘要:

视觉-语言模型(VLMs)正在越来越多地应用于现实世界和具身环境中,其中安全决策依赖于视觉上下文。然而,仍不清楚哪些视觉证据驱动这些判断。我们研究多模态安全行为在VLMs中是否能通过简单的语义线索引导。我们介绍了一个语义引导框架,该框架应用受控的文本、视觉和认知干预,而无需改变底层场景内容。为了评估这些影响,我们提出了SAVeS,一个在语义线索下的情境安全基准,以及一个评估协议,该协议将行为拒绝、基于语境的安全推理和虚假拒绝分开。在多个VLMs和一个额外的最先进基准上的实验表明,安全决策对语义线索高度敏感,表明依赖于学习的视觉-语言关联,而不是基于视觉的理解。我们进一步证明,自动化引导流程可以利用这些机制,突显了多模态安全系统中的潜在漏洞。

英文摘要:

Vision-language models (VLMs) are increasingly deployed in real-world and embodied settings where safety decisions depend on visual context. However, it remains unclear which visual evidence drives these judgments. We study whether multimodal safety behavior in VLMs can be steered by simple semantic cues. We introduce a semantic steering framework that applies controlled textual, visual, and cognitive interventions without changing the underlying scene content. To evaluate these effects, we propose SAVeS, a benchmark for situational safety under semantic cues, together with an evaluation protocol that separates behavioral refusal, grounded safety reasoning, and false refusals. Experiments across multiple VLMs and an additional state-of-the-art benchmark show that safety decisions are highly sensitive to semantic cues, indicating reliance on learned visual-linguistic associations rather than grounded visual understanding. We further demonstrate that automated steering pipelines can exploit these mechanisms, highlighting a potential vulnerability in multimodal safety systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑