Re-Align：通过检索增强的直接偏好优化对齐视觉语言模型
Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization
AI总结：
本文提出Re-Align框架，通过图像检索构建双偏好数据集并引入rDPO视觉偏好目标，有效缓解视觉语言模型幻觉并提升VQA性能。
摘要：
大型视觉语言模型（VLMs）的出现通过整合视觉模态，拓宽了单模态大型语言模型（LLMs）的范围和能力，从而在多种现实场景中解锁了变革性的跨模态应用。尽管VLMs表现出色，但它们容易出现严重的幻觉，尤其是跨模态不一致的形式。基于从人类反馈中进行强化学习（RLHF）在LLMs对齐方面的成功，近期进展集中在将直接偏好优化（DPO）应用于精心整理的数据集上，以缓解这些问题。然而，这类方法通常以蛮力方式引入偏好信号，忽视了视觉信息在对齐过程中的关键作用。在本文中，我们提出了Re-Align，一种新颖的对齐框架，利用图像检索构建双偏好数据集，有效融合文本和视觉偏好信号。我们进一步提出了rDPO，这是标准直接偏好优化的扩展，在微调过程中加入了额外的视觉偏好目标。我们的实验结果表明，Re-Align不仅比先前方法更有效地缓解幻觉，还在通用视觉问答（VQA）任务中带来了显著的性能提升。此外，我们表明Re-Align在广泛的VLM规模和架构范围内保持了鲁棒性和可扩展性。这项工作代表了多模态LLMs对齐方面的重要进展，为更可靠和有效的跨模态应用铺平了道路。我们在https://github.com/taco-group/Re-Align中发布了所有代码。
英文摘要：
The emergence of large Vision Language Models (VLMs) has broadened the scope and capabilities of single-modal Large Language Models (LLMs) by integrating visual modalities, thereby unlocking transformative cross-modal applications in a variety of real-world scenarios. Despite their impressive performance, VLMs are prone to significant hallucinations, particularly in the form of cross-modal inconsistencies. Building on the success of Reinforcement Learning from Human Feedback (RLHF) in aligning LLMs, recent advancements have focused on applying direct preference optimization (DPO) on carefully curated datasets to mitigate these issues. Yet, such approaches typically introduce preference signals in a brute-force manner, neglecting the crucial role of visual information in the alignment process. In this paper, we introduce Re-Align, a novel alignment framework that leverages image retrieval to construct a dual-preference dataset, effectively incorporating both textual and visual preference signals. We further introduce rDPO, an extension of the standard direct preference optimization that incorporates an additional visual preference objective during fine-tuning. Our experimental results demonstrate that Re-Align not only mitigates hallucinations more effectively than previous methods but also yields significant performance gains in general visual question-answering (VQA) tasks. Moreover, we show that Re-Align maintains robustness and scalability across a wide range of VLM sizes and architectures. This work represents a significant step forward in aligning multimodal LLMs, paving the way for more reliable and effective cross-modal applications. We release all the code in https://github.com/taco-group/Re-Align.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
