利用策略数据优化LVLMs以实现有效的幻觉缓解
Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigation
AI总结：
本文提出利用策略数据优化LVLMs，通过幻觉分类器和动态加权DPO算法，显著降低幻觉率并提升模型性能。
摘要：
近期，大型视觉-语言模型（LVLMs）已升至多模态任务的有前景的方法。然而，系统性的幻觉缓解仍然是一个关键挑战。在本工作中，我们首先分析了LVLM幻觉缓解的数据生成过程，并确认策略数据显著优于非策略数据，因此需要高效且可靠的策略数据偏好标注。我们指出，现有标注方法会在训练样本中引入额外的幻觉，可能会增强模型的幻觉模式，为了解决这个问题，我们提出训练一个幻觉分类器，给出二元标注，这保证了后续对齐的干净样本。为了进一步利用策略数据的潜力，我们设计了一种稳健的迭代直接偏好优化（DPO）算法，采用动态样本重新加权方案。我们在三个基准上进行了全面的实验，并与8种最先进的基线进行比较。特别是，我们的方法将LLaVA-1.5-7B在MMHalBench上的幻觉率减少了50.8%，在Object HalBench上的平均幻觉率减少了79.5%；更显著的是，我们的方法充分挖掘了开源模型的潜力，使LLaVA-1.5-13B甚至超过了GPT-4V的性能。
英文摘要：
Recently, large vision-language models (LVLMs) have risen to be a promising approach for multimodal tasks. However, principled hallucination mitigation remains a critical challenge.In this work, we first analyze the data generation process in LVLM hallucination mitigation and affirm that on-policy data significantly outperforms off-policy data, which thus calls for efficient and reliable preference annotation of on-policy data. We then point out that, existing annotation methods introduce additional hallucination in training samples, which may enhance the model's hallucination patterns, to address this problem, we propose training a hallucination classifier giving binary annotations, which guarantee clean chosen samples for the subsequent alignment. To further harness of the power of on-policy data, we design a robust iterative direct preference optimization (DPO) algorithm adopting a dynamic sample reweighting scheme. We conduct comprehensive experiments on three benchmarks with comparison to 8 state-of-the-art baselines. In particular, our approach reduces the hallucination rate of LLaVA-1.5-7B on MMHalBench by 50.8% and the average hallucination rate on Object HalBench by 79.5%; more significantly, our method fully taps into the potential of open-source models, enabling LLaVA-1.5-13B to even surpass the performance of GPT-4V.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
