通过DPO缓解大型视觉语言模型中的幻觉：策略内数据是关键
Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key
AI总结：
针对LVLMs的幻觉问题，提出OPA-DPO框架，利用专家反馈纠正幻觉响应并以策略内方式对齐数据，仅用4.8k数据即在AMBER和Object-Hal基准上显著超越使用16k样本的SOTA算法。
摘要：
幻觉仍然是大型视觉语言模型（LVLMs）面临的一项重大挑战。直接偏好优化（DPO）作为一种解决幻觉问题的简单方案，受到了越来越多的关注。它直接从构建的偏好对中学习，这些偏好对反映了在相同提示和图像下响应中幻觉的严重程度。然而，现有工作中的不同数据构建方法带来了显著的性能差异。我们在此确定了一个关键因素：结果在很大程度上取决于构建的数据是否相对于DPO的初始（参考）策略在策略上保持一致。理论分析表明，从非策略数据中学习会受到更新策略与参考策略之间KL散度的阻碍。从数据集分布的角度，我们系统地总结了现有采用DPO解决幻觉问题的算法的固有缺陷。为了缓解这些问题，我们提出了策略内对齐（OPA）-DPO框架，该框架独特地利用专家反馈来纠正幻觉响应，并以策略内方式同时对原始响应和专家修订后的响应进行对齐。值得注意的是，仅使用4.8k数据，OPA-DPO在LLaVA-1.5-7B上的幻觉率实现了额外降低：与使用16k样本训练的先前SOTA算法相比，在AMBER基准上降低了13.26%，在Object-Hal基准上降低了5.39%。我们的实现可在https://github.com/zhyang2226/OPA-DPO获取。
英文摘要：
Hallucination remains a major challenge for Large Vision-Language Models (LVLMs). Direct Preference Optimization (DPO) has gained increasing attention as a simple solution to hallucination issues. It directly learns from constructed preference pairs that reflect the severity of hallucinations in responses to the same prompt and image. Nonetheless, different data construction methods in existing works bring notable performance variations. We identify a crucial factor here: outcomes are largely contingent on whether the constructed data aligns on-policy w.r.t the initial (reference) policy of DPO. Theoretical analysis suggests that learning from off-policy data is impeded by the presence of KL-divergence between the updated policy and the reference policy. From the perspective of dataset distribution, we systematically summarize the inherent flaws in existing algorithms that employ DPO to address hallucination issues. To alleviate the problems, we propose On-Policy Alignment (OPA)-DPO framework, which uniquely leverages expert feedback to correct hallucinated responses and aligns both the original and expert-revised responses in an on-policy manner. Notably, with only 4.8k data, OPA-DPO achieves an additional reduction in the hallucination rate of LLaVA-1.5-7B: 13.26% on the AMBER benchmark and 5.39% on the Object-Hal benchmark, compared to the previous SOTA algorithm trained with 16k samples. Our implementation is available at https://github.com/zhyang2226/OPA-DPO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
