PEA-DPO：用于多模态大语言模型对齐的感知增强型直接偏好优化
PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment
浏览论文内容
中文总结 AI 辅助
针对多模态大语言模型偏好优化存在的视觉不敏感性问题，提出PEA-DPO框架，利用视觉偏好信号缓解该问题，提升多模态对齐效果并减少幻觉。
中文摘要 AI 辅助
直接偏好优化（DPO）已成为将大语言模型（LLMs）与人类偏好对齐的有效方法，但其在多模态场景中的应用尚未被探索。通过表征分析，我们发现多模态偏好优化存在一个关键局限，我们称之为视觉不敏感性：模型常无法区分图像与移除关键视觉上下文后的图像。我们的理论分析进一步揭示该问题的两种表现形式，即跨图像不敏感性和图像内不敏感性。为应对这些挑战，我们提出用于多模态大语言模型对齐的框架——感知增强型对齐DPO（PEA-DPO），其明确利用视觉偏好信号以克服视觉不敏感性。我们还提供理论分析证明PEA-DPO可缓解这两种失效模式。实验结果表明，PEA-DPO在保留基础模型语言建模能力的同时，增强了对视觉上下文的敏感性；在不同规模多模态大语言模型的三个幻觉基准上的评估显示，PEA-DPO可有效缓解视觉不敏感性，实现更强的多模态对齐，并大幅减少幻觉。
英文摘要
Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its adaptation to multimodal settings remains unexplored. Through representational analysis, we identify a key limitation in multimodal preference optimization, which we term visual insensitivity: models often fail to distinguish between images and those with critical visual context removed. Our theoretical analysis further uncovers two manifestations of this problem, namely Across-Image Insensitivity and Within-Image Insensitivity. To address these challenges, we propose Perception-Enhanced Alignment DPO (PEA-DPO), a framework for multimodal LLMs alignment, which explicitly leverages visual preference signals to overcome visual insensitivity. We further provide a theoretical analysis demonstrating that PEA-DPO provably mitigates both failure modes. Empirical results demonstrate that PEA-DPO enhances sensitivity to visual context while preserving the language modeling capacity of the base model. Evaluations across three hallucination benchmarks using MLLMs of varying scales show that PEA-DPO effectively mitigates visual insensitivity, achieves stronger multimodal alignment, and substantially reduces hallucinations.
发表机构
- University of Science and Technology of China（中国科学技术大学）
- National University of Singapore（新加坡国立大学）
机构由 AI 辅助整理，请以论文原文为准。
