arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.04736cs.CL
收藏

AM$^3$Safety: 向多模态多轮安全对齐的数据高效方法

Towards Multi-modal Multi-turn Safety: From Agentic Interaction to Strategic Alignment

Hong Kong University of Science and Technology(香港科技大学);Zhongshan School of Medicine, SUN YAT-SEN UNIVERSITY(中山医学院,孙中山大学);University of Edinburgh(爱丁堡大学);University of Washington(华盛顿大学)

Han Zhu, Jiale Chen, Chengkun Cai, Shengjie Sun, Haoran Li, Yujin Zhou, Chi-Min Chan, Pengcheng Wen, Lei Li, Yike Guo, Sirui Han

AI总结:

AM$^3$Safety通过结合冷启动拒绝阶段和组相对策略优化,有效提升多模态多轮对话的安全性,降低攻击成功率并增强模型的无害与帮助维度。

摘要:

多模态大语言模型(MLLMs)正在越来越多地应用于交互式应用。然而,它们的安全漏洞在多轮多模态场景中变得尤为突出,其中有害意图可以在多个回合中逐步重建,而安全协议在对话过程中逐渐消失。现有的基于人类反馈的强化学习(RLHF)对齐方法大多针对单轮视觉问答(VQA)任务开发,通常需要昂贵的手动偏好注释,限制了它们在对话中的有效性和可扩展性。为了解决这一挑战,我们提出了InterSafe-V,一个开源的多模态对话数据集,包含11,270个对话和500个特别设计的拒绝VQA样本。该数据集通过多个模型之间的交互构建,旨在更准确地反映现实世界场景,并包含针对特定领域的专门VQA对。基于此数据集,我们提出了AM$^3$Safety,一种结合冷启动拒绝阶段和组相对策略优化(GRPO)微调的框架,使用跨整个对话的轮次感知双目标奖励。在Qwen2.5-VL-7B-Instruct和LLaVA-NeXT-7B上的实验显示,攻击成功率(ASR)下降超过10%,同时在多模态多轮安全基准上,MLLMs的无害维度增加了至少8%,有帮助维度增加了超过13%,同时保持了其通用能力。

英文摘要:

Despite remarkable capability in multi-modal understanding, deploying Multi-modal Large Language Models (MLLMs) in open-ended conversational scenarios introduces safety risks that remain poorly addressed by existing alignment methods. Unlike simple malicious visual question and answer (VQA) pairs , multi-turn interactions enable adversaries to incrementally reconstruct harmful intent across dialogues, progressively bypassing safety constraints in ways that are difficult to detect at any individual turn. Meanwhile, conventional reinforcement learning from human feedback (RLHF) approaches are unsuitable for this situation: designed primarily for VQA tasks, they neither capture cross-turn risk dynamics nor scale efficiently without costly manual preference annotation. To close this gap, we introduce \textbf{MINT-Safe}, an open-source visual multi-turn training dataset comprising 11,270 multi-image dialogues and 500 refusal VQA pairs, constructed via multi-agent interaction with text-to-image (T2I) tool-call augmentation. Building on MINT-Safe, we propose \textbf{TAD-Align}, a dialogue safety alignment framework centered on a turn-aware dual-objective reward function. Rather than treating all dialogue turns uniformly, TAD-Align leverages rollout-based safety score variance to dynamically identify turns where the model exhibits inconsistent safety behavior, and adaptively up-weights these turns during optimization. Experiments on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-7B demonstrate reductions of over 10\% in Attack Success Rate (ASR), alongside improvements of at least 8\% in harmlessness and 13\% in helpfulness on multi-modal multi-turn safety benchmarks, while preserving general model capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑