利用 Blind Preference Optimization 实现无害多模态助手
Towards Harmless Multimodal Assistants with Blind Preference Optimization
AI总结：
针对多模态大语言模型安全对齐缺乏偏好数据的问题，本文构建 MMSafe-PO 数据集并提出 BPO 方法，显著提升安全性且具有跨基准鲁棒性。
摘要：
多模态大语言模型（MLLMs）已在多模态理解、推理和交互方面展现出令人瞩目的能力。鉴于 MLLMs 的广泛应用，相关安全问题正变得日益关键。由于偏好优化在使 MLLMs 与人类偏好对齐方面具有显著效果，目前迫切需要面向 MLLMs 的安全相关偏好数据。为解决这一问题，我们构建了面向无害多模态助手的 MMSafe-PO 偏好数据集，其特点是包含多模态指令、对话格式以及来自人类反馈的排序成对回复。我们还发现了两个具有启发性的观察结果：模态协同防御和模态欺骗，它们表明 MLLMs 具备一定程度的内在防御能力，同时仍呈现出独特的安全挑战。基于这些观察，我们提出了 Blind Preference Optimization（BPO）方法。在三个基准上开展的综合实验表明，BPO 有效增强了 MLLMs 的安全能力。值得注意的是，BPO 将基础 MLLM 的安全率显著提升了 45.0%，优于 DPO 方法。此外，将 BPO 应用于 MMSafe-PO 数据集后，基础 MLLM 在其他安全基准上的不安全率大幅降低（MM-SafetyBench 上降低 14.5%，HarmEval 上降低 82.9%），证明了该数据集和方法的有效性与鲁棒性。我们在 https://lu-yang666.github.io/MMsafe-PO-Web/ 发布代码和数据。
英文摘要：
Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in multimodal understanding, reasoning, and interaction. Given the extensive applications of MLLMs, the associated safety issues have become increasingly critical. Due to the effectiveness of preference optimization in aligning MLLMs with human preferences, there is an urgent need for safety-related preference data for MLLMs. To address this, we construct the MMSafe-PO preference dataset towards harmless multimodal assistants, featuring multimodal instructions, the conversational format, and ranked paired responses from human feedback. We also identify two insightful observations: modality co-defense and modality cheating, which illustrate that MLLMs possess a certain level of inherent defense while still presenting unique safety challenges. Based on these observations, we propose the Blind Preference Optimization (BPO) approach. Comprehensive experiments on three benchmarks show that BPO effectively enhances the safety capabilities of MLLMs. Notably, BPO significantly improves the safety rate of the base MLLM by 45.0%, outperforming the DPO approach. Additionally, applying BPO to the MMSafe-PO dataset greatly reduces the base MLLM's unsafe rate on other safety benchmarks (14.5% on MM-SafetyBench and 82.9% on HarmEval, demonstrating the effectiveness and robustness of both the dataset and the approach. We release code and data at https://lu-yang666.github.io/MMsafe-PO-Web/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
