TLDR:面向大型视觉语言模型的令牌级侦探奖励模型
TLDR: Token-Level Detective Reward Model for Large Vision Language Models
AI总结:
本文提出令牌级侦探奖励模型TLDR,通过扰动生成难负样本及令牌级标签,提供细粒度反馈,提升多模态模型性能并加速人工标注3倍。
摘要:
尽管奖励模型在改进多模态大语言模型方面取得了成功,但奖励模型本身仍然粗糙且包含的信息极少。值得注意的是,现有的奖励模型仅通过为任何文本(无论文本多长)分配一个二元反馈来模拟人类标注。在多模态语言模型领域,模型需要同时处理图像和文本,一个朴素的奖励模型可能会学习到对文本的隐式偏见,从而对图像的依赖减弱。在本文中,我们提出了一种令牌级侦探奖励模型(TLDR),为每个文本令牌提供细粒度的标注。我们首先引入一种基于扰动的方法来生成合成难负样本及其令牌级标签,以训练TLDR模型。然后,我们展示了TLDR模型在辅助现成模型自我纠正其生成内容以及作为幻觉评估工具方面的丰富实用性。我们表明,TLDR自动训练令牌级似然优化,并能显著提升基础模型的性能。最后,我们证明TLDR模型可以将人类标注速度提高3倍,从而获取更广泛的高质量视觉语言数据。
英文摘要:
Although reward models have been successful in improving multimodal large language models, the reward models themselves remain brutal and contain minimal information. Notably, existing reward models only mimic human annotations by assigning only one binary feedback to any text, no matter how long the text is. In the realm of multimodal language models, where models are required to process both images and texts, a naive reward model may learn implicit biases toward texts and become less grounded in images. In this paper, we propose a $\textbf{T}$oken-$\textbf{L}$evel $\textbf{D}$etective $\textbf{R}$eward Model ($\textbf{TLDR}$) to provide fine-grained annotations to each text token. We first introduce a perturbation-based method to generate synthetic hard negatives and their token-level labels to train TLDR models. Then we show the rich usefulness of TLDR models both in assisting off-the-shelf models to self-correct their generations, and in serving as a hallucination evaluation tool. We show that TLDR automatically trains a token-level likelihood optimization, and can improve the base model's performance significantly. Finally, we show that TLDR models can significantly speed up human annotation by 3 times to acquire a broader range of high-quality vision language data.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。