DUAL-Bench: 测量视觉语言模型中的过度拒绝与鲁棒性
DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models
AI总结：
本文提出DUAL-Bench，通过评估18种VLM在12类危险场景下的表现，揭示模型在双重使用场景中的脆弱安全边界，强调安全完成与过度拒绝的平衡需求。
摘要：
随着视觉语言模型（VLMs）能力不断增强，保持安全与有用之间的平衡成为核心挑战。安全机制虽然必要，但可能引发过度拒绝，即模型因过度谨慎而拒绝良性请求。目前缺乏系统性评估视觉模态中过度拒绝的基准。本文提出DUAL-Bench，一个大规模多模态基准，聚焦于VLMs的过度拒绝和安全完成。我们评估了18种VLM在语义保持的视觉扰动下，12类危险类别中的表现。在双重使用场景中，模型的安全边界极为脆弱，陷入二元陷阱：要么过于敏感地直接拒绝，要么产生危险内容。即使表现最佳的GPT-5-Nano模型，安全完成率也仅为12.9%，而GPT-5和Qwen家族平均为7.9%和3.9%。我们希望DUAL-Bench促进更细致的对齐策略，平衡多模态安全与效用。内容警示：本文包含敏感和潜在危险内容。
英文摘要：
As vision-language models (VLMs) become increasingly capable, maintaining a balance between safety and usefulness remains a central challenge. Safety mechanisms, while essential, can backfire, causing over-refusal, where models decline benign requests out of excessive caution. Yet, there is currently a significant lack of benchmarks that have systematically addressed over-refusal in the visual modality. This setting introduces unique challenges, such as dual-use cases where an instruction is harmless, but the accompanying image contains harmful content. Models frequently fail in such scenarios, either refusing too conservatively or completing tasks unsafely, which highlights the need for more fine-grained alignment. The ideal behaviour is safe completion, i.e., fulfilling the benign parts of a request while explicitly warning about any potentially harmful elements. To address this, we present DUAL-Bench, a large scale multimodal benchmark focused on over-refusal and safe completion in VLMs. We evaluated 18 VLMs across 12 hazard categories under semantics-preserving visual perturbations. In dual-use scenarios, models exhibit extremely fragile safety boundaries. They fall into a binary trap: either overly sensitive direct refusal or defenseless generation of dangerous content. Consequently, even the best-performing model GPT-5-Nano, at just 12.9% safe completion, with GPT-5 and Qwen families averaging 7.9% and 3.9%. We hope DUAL-Bench fosters nuanced alignment strategies balancing multimodal safety and utility. Content Warning: This paper contains examples of sensitive and potentially hazardous content.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
