系统性奖励缺口优化以缓解视觉语言模型的幻觉
Systematic Reward Gap Optimization for Mitigating VLM Hallucinations
AI总结：
TPR通过主题级偏好重写系统优化奖励缺口配置，显著减少VLM幻觉并提升对齐效果。
摘要：
直接偏好优化（DPO）在缓解视觉语言模型（VLM）幻觉方面的成功关键取决于偏好对内的真实奖励缺口。然而，当前方法通常依赖于排序或重写策略，在数据整理过程中往往难以系统地优化这些奖励缺口。核心困难在于精确地表征和战略性地操控整体奖励缺口配置，即在数据中每个偏好对内如何塑造这些奖励缺口。为了解决这一问题，我们引入了主题级偏好重写（TPR），一种新的框架，用于系统地优化奖励缺口配置。通过选择性地将VLM响应中的语义主题替换为模型自身重新采样的候选者进行针对性重写，TPR可以在细粒度语义细节上提供主题级控制。这种精确控制使高级数据整理策略成为可能，例如逐步调整被拒绝响应的难度，从而塑造出有效的奖励缺口配置，引导模型克服具有挑战性的幻觉。全面的实验表明，TPR在多个幻觉基准测试中实现了最先进的性能，比先前方法平均高出20%。值得注意的是，它在ObjectHal-Bench上显著减少了幻觉，高达93%，并且在稳健且成本效益高的VLM对齐方面也表现出色。代码和数据集可在https://tpr-dpo.github.io上获得。
英文摘要：
The success of Direct Preference Optimization (DPO) in mitigating hallucinations in Vision Language Models (VLMs) critically hinges on the true reward gaps within preference pairs. However, current methods, typically relying on ranking or rewriting strategies, often struggle to optimize these reward gaps in a systematic way during data curation. A core difficulty lies in precisely characterizing and strategically manipulating the overall reward gap configuration, that is, the deliberate design of how to shape these reward gaps within each preference pair across the data. To address this, we introduce Topic-level Preference Rewriting(TPR), a novel framework designed for the systematic optimization of reward gap configuration. Through selectively replacing semantic topics within VLM responses with model's own resampled candidates for targeted rewriting, TPR can provide topic-level control over fine-grained semantic details. This precise control enables advanced data curation strategies, such as progressively adjusting the difficulty of rejected responses, thereby sculpting an effective reward gap configuration that guides the model to overcome challenging hallucinations. Comprehensive experiments demonstrate TPR achieves state-of-the-art performance on multiple hallucination benchmarks, outperforming previous methods by an average of 20%. Notably, it significantly reduces hallucinations by up to 93% on ObjectHal-Bench, and also exhibits superior data efficiency towards robust and cost-effective VLM alignment. Code and datasets are available at https://tpr-dpo.github.io .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
