arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.12109cs.CVcs.CL
收藏

RoVRM:通过辅助文本偏好数据优化的鲁棒视觉奖励模型

RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data

Northeastern University(东北大学);NiuTrans Research(牛翻译研究院);Institute of Psychology, CAS(中国科学院心理研究所)

Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Murun Yang, Qiaozhi He, Tong Xiao, Chunliang Zhang, Tongran Liu, Quan Du, Di Yang, Jingbo Zhu

AI总结:

本文提出RoVRM,通过三阶段渐进式训练和基于最优传输的偏好数据选择,利用辅助文本偏好数据缓解视觉偏好数据稀缺问题,有效提升了LVLMs的人类偏好对齐效果。

摘要:

大型视觉语言模型(LVLMs)通常难以与人类偏好对齐,导致在没有适当视觉上下文的情况下生成误导性内容(也称为幻觉)等问题。解决该问题的一个有前景的方案是使用人类偏好对齐技术,例如best-of-n采样和强化学习。然而,这些技术面临着视觉偏好数据稀缺的困难,而训练视觉奖励模型(VRM)需要这些数据。在本研究中,我们延续了这一研究方向。我们提出了一个鲁棒视觉奖励模型(RoVRM),以改善LVLMs的人类偏好对齐。RoVRM通过三阶段渐进式训练和基于最优传输的偏好数据选择来利用辅助文本偏好数据,从而有效缓解视觉偏好数据的稀缺问题。我们在基于LLaVA-1.5-7B和-13B模型的常用视觉语言任务上对RoVRM进行了实验。实验结果表明,RoVRM始终优于传统的VRMs。此外,我们的三阶段渐进式训练和偏好数据选择方法能够比基于排序的对齐技术(如direct preference optimization)产生一致的性能提升。

英文摘要:

Large vision-language models (LVLMs) often fail to align with human preferences, leading to issues like generating misleading content without proper visual context (also known as hallucination). A promising solution to this problem is using human-preference alignment techniques, such as best-of-n sampling and reinforcement learning. However, these techniques face the difficulty arising from the scarcity of visual preference data, which is required to train a visual reward model (VRM). In this work, we continue the line of research. We present a Robust Visual Reward Model (RoVRM) which improves human-preference alignment for LVLMs. RoVRM leverages auxiliary textual preference data through a three-phase progressive training and optimal transport-based preference data selection to effectively mitigate the scarcity of visual preference data. We experiment with RoVRM on the commonly used vision-language tasks based on the LLaVA-1.5-7B and -13B models. Experimental results demonstrate that RoVRM consistently outperforms traditional VRMs. Furthermore, our three-phase progressive training and preference data selection approaches can yield consistent performance gains over ranking-based alignment techniques, such as direct preference optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑