arXiv 2501.166292025-01-28cs.CLcs.CV

CHiP：面向多模态 LLM 的跨模态分层直接偏好优化

CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs

National University of Singapore（新加坡国立大学）；Fudan University（复旦大学）；Digital Twin Institute（数字孪生研究院）；Eastern Institute of Technology（东方理工学院）

Jinlan Fu, Shenzhen Huangfu, Hao Fei, Xiaoyu Shen, Bryan Hooi, Xipeng Qiu, See-Kiong Ng

备注：: Accepted by ICLR 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对多模态大模型在 DPO 中难以对齐图文表征及区分幻觉的问题，提出跨模态分层直接偏好优化方法 CHiP，通过引入视觉偏好和多粒度文本偏好优化模块，在多个基准上显著降低了幻觉。

摘要：

尽管多模态大语言模型（MLLMs）具备令人印象深刻的能力，但仍然受幻觉问题的困扰。近期研究试图通过使用基于文本响应的偏好对，将直接偏好优化（DPO）应用于多模态场景以缓解此问题。然而，我们对表征分布的分析表明，多模态 DPO 难以对齐图像与文本表征，也难以区分幻觉与非幻觉描述。为了应对这些挑战，在本研究中，我们提出了跨模态分层直接偏好优化（CHiP）以解决这些局限性。我们在 DPO 框架内引入了视觉偏好优化模块，使 MLLMs 能够同时从文本和视觉偏好中学习。此外，我们提出了一个分层文本偏好优化模块，允许模型在多个粒度级别上捕获偏好，包括响应、片段和 token 级别。我们通过定量和定性分析评估了 CHiP，在多个基准上的结果证明了其在减少幻觉方面的有效性。在 Object HalBench 数据集上，CHiP 在减少幻觉方面优于 DPO，分别基于基础模型 Muffin 和 LLaVA 模型实现了 52.7% 和 55.5% 的相对百分点提升。我们公开了所有数据集和代码：https://github.com/LVUGAI/CHiP。

英文摘要：

Multimodal Large Language Models (MLLMs) still struggle with hallucinations despite their impressive capabilities. Recent studies have attempted to mitigate this by applying Direct Preference Optimization (DPO) to multimodal scenarios using preference pairs from text-based responses. However, our analysis of representation distributions reveals that multimodal DPO struggles to align image and text representations and to distinguish between hallucinated and non-hallucinated descriptions. To address these challenges, in this work, we propose a Cross-modal Hierarchical Direct Preference Optimization (CHiP) to address these limitations. We introduce a visual preference optimization module within the DPO framework, enabling MLLMs to learn from both textual and visual preferences simultaneously. Furthermore, we propose a hierarchical textual preference optimization module that allows the model to capture preferences at multiple granular levels, including response, segment, and token levels. We evaluate CHiP through both quantitative and qualitative analyses, with results across multiple benchmarks demonstrating its effectiveness in reducing hallucinations. On the Object HalBench dataset, CHiP outperforms DPO in hallucination reduction, achieving improvements of 52.7% and 55.5% relative points based on the base model Muffin and LLaVA models, respectively. We make all our datasets and code publicly available: https://github.com/LVUGAI/CHiP.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。