基于自校准视觉锚定奖励的Token偏好优化以缓解幻觉问题
Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigation
AI总结：
针对现有方法缺乏可扩展Token级奖励和忽略视觉锚定Token的缺陷，提出TPO模型，通过自校准视觉锚定Token级奖励及视觉感知训练目标，在LLAVA-1.5-7B上实现幻觉基准性能绝对提升。
摘要：
直接偏好优化（DPO）已被证明能通过更紧密地对齐大型视觉语言模型（LVLMs）的输出与人类偏好，有效缓解其幻觉问题。尽管近期取得进展，但现有方法存在两个缺陷：1）缺乏可扩展的Token级奖励；2）忽略视觉锚定Token。为此，我们提出一种具有自校准奖励的新型Token偏好优化模型（称为TPO），无需细粒度标注即可自适应关注视觉相关Token。具体而言，我们引入Token级“视觉锚定奖励”，即生成Token在原始图像和损坏图像条件下的逻辑分布差异。此外，为突出信息丰富的视觉锚定Token，我们提出视觉感知训练目标以增强更准确的Token级优化。大量实验结果表明，所提TPO取得了最先进的性能，例如基于LLAVA-1.5-7B时，在幻觉基准测试中实现了性能的绝对提升。
英文摘要：
Direct Preference Optimization (DPO) has been demonstrated to be highly effective in mitigating hallucinations in Large Vision Language Models (LVLMs) by aligning their outputs more closely with human preferences. Despite the recent progress, existing methods suffer from two drawbacks: 1) Lack of scalable token-level rewards; and 2) Neglect of visual-anchored tokens. To this end, we propose a novel Token Preference Optimization model with self-calibrated rewards (dubbed as TPO), which adaptively attends to visual-correlated tokens without fine-grained annotations. Specifically, we introduce a token-level \emph{visual-anchored} \emph{reward} as the difference of the logistic distributions of generated tokens conditioned on the raw image and the corrupted one. In addition, to highlight the informative visual-anchored tokens, a visual-aware training objective is proposed to enhance more accurate token-level optimization. Extensive experimental results have manifested the state-of-the-art performance of the proposed TPO. For example, by building on top of LLAVA-1.5-7B, our TPO boosts the performance absolute improvement for hallucination benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
