面向多负样本多模态直接偏好优化的重要性采样
Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization
AI总结：
针对现有多模态DPO依赖单负样本成对比较导致优化偏差与幻觉的问题，提出MISP-DPO框架，通过Plackett-Luce模型结合语义感知多负样本采样与重要性采样，在五个基准上持续提升多模态对齐效果。
摘要：
直接偏好优化（DPO）近期已从纯文本模型扩展至视觉语言模型。然而现有方法依赖过度简化的成对比较，通过基础扰动或基于相似度的检索生成单个负样本图像，无法捕捉多模态偏好的复杂本质，会引发优化偏差与幻觉问题。为解决该问题，我们提出MISP-DPO，这是首个通过Plackett-Luce模型将多个语义多样的负样本图像融入多模态DPO的框架。我们的方法将提示词与候选图像嵌入到CLIP（对比语言-图像预训练）空间中，并应用稀疏自编码器将语义偏差拆解为可解释因子。负样本基于重构难度、与正样本的语义偏差以及相互多样性进行选择，从而提供更广泛、更具信息量的监督信号。为处理多负样本比较，我们采用Plackett-Luce目标，并引入一种提升训练效率的重要性采样策略。在五个不同基准上的实验表明，MISP-DPO相较于现有方法持续提升了多模态对齐效果，验证了语义感知的多负样本采样在基于偏好的学习中的有效性。
英文摘要：
Direct Preference Optimization (DPO) has recently been extended from text-only models to vision-language models. However, existing methods rely on oversimplified pairwise comparisons, generating a single negative image via basic perturbations or similarity-based retrieval, which fail to capture the complex nature of multimodal preferences, inducing optimization bias and hallucinations. To address this issue, we propose MISP-DPO, the first framework to incorporate multiple, semantically diverse negative images in multimodal DPO via the Plackett-Luce model. Our method embeds prompts and candidate images in CLIP (Contrastive Language-Image Pretraining) space and applies a sparse autoencoder to uncover semantic deviations into interpretable factors. Negative samples are selected based on reconstruction difficulty, semantic deviation from the positive, and mutual diversity, yielding broader and more informative supervision. To handle multi-negative comparisons, we adopt a Plackett-Luce objective and introduce an importance sampling strategy that improves training efficiency. Experiments across five diverse benchmarks demonstrate that MISP-DPO consistently improves multimodal alignment over prior methods, validating the effectiveness of semantic-aware, multi-negative sampling in preference-based learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
