arXiv 2601.006232026-01-02cs.AI

DA-DPO：面向减少多模态大语言模型幻觉的高效难度感知偏好优化

DA-DPO: Cost-efficient Difficulty-aware Preference Optimization for Reducing MLLM Hallucinations

ShanghaiTech University（上海科技大学）；Lingang Laboratory（灵冈实验室）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）

Longtian Qiu, Shan Ning, Chuyu Zhang, Jiaxuan Sun, Xuming He

备注：: Accepted by TMLR

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

DA-DPO通过难度感知机制优化多模态大语言模型的偏好学习，有效减少幻觉并提升模型鲁棒性与泛化能力。

摘要：

直接偏好优化（DPO）在缓解多模态大语言模型（MLLM）的幻觉方面展现出了强大的潜力。然而，现有的多模态DPO方法常常由于偏好数据中的难度失衡而出现过拟合问题。我们的分析表明，MLLM倾向于过度强调容易区分的偏好对，这阻碍了细粒度幻觉抑制并降低了整体性能。为了解决这一问题，我们提出了难度感知直接偏好优化（DA-DPO），一种成本效益高的框架，旨在平衡学习过程。DA-DPO包含两个主要组成部分：（1）难度估计利用预训练的视觉-语言模型，结合互补的生成和对比目标，其输出通过分布感知的投票策略整合，以产生稳健的难度评分，而无需额外训练；（2）难度感知训练根据估计的难度重新加权偏好对，降低简单样本的权重同时强调更困难的样本以缓解过拟合。该框架通过优先考虑具有挑战性的示例实现了更有效的偏好优化，而无需新的数据或额外的微调阶段。广泛的实验表明，DA-DPO在多模态偏好优化方面始终表现出色，产生了更强的抗幻觉鲁棒性，并在标准基准上实现了更好的泛化能力，同时保持了计算效率。项目页面可在https://artanic30.github.io/project_pages/DA-DPO/上找到。

英文摘要：

Direct Preference Optimization (DPO) has shown strong potential for mitigating hallucinations in Multimodal Large Language Models (MLLMs). However, existing multimodal DPO approaches often suffer from overfitting due to the difficulty imbalance in preference data. Our analysis shows that MLLMs tend to overemphasize easily distinguishable preference pairs, which hinders fine-grained hallucination suppression and degrades overall performance. To address this issue, we propose Difficulty-Aware Direct Preference Optimization (DA-DPO), a cost-effective framework designed to balance the learning process. DA-DPO consists of two main components: (1) Difficulty Estimation leverages pre-trained vision--language models with complementary generative and contrastive objectives, whose outputs are integrated via a distribution-aware voting strategy to produce robust difficulty scores without additional training; and (2) Difficulty-Aware Training reweights preference pairs based on their estimated difficulty, down-weighting easy samples while emphasizing harder ones to alleviate overfitting. This framework enables more effective preference optimization by prioritizing challenging examples, without requiring new data or extra fine-tuning stages. Extensive experiments demonstrate that DA-DPO consistently improves multimodal preference optimization, yielding stronger robustness to hallucinations and better generalization across standard benchmarks, while remaining computationally efficient. The project page is available at https://artanic30.github.io/project_pages/DA-DPO/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。