聚焦视觉关键点:通过视觉信息增益进行大视觉语言模型的定向训练
Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain
AI总结:
本文提出通过视觉信息增益(VIG)指标,对大视觉语言模型进行定向训练,以提升视觉基础性并减少语言偏见,通过优先选择高VIG样本和token来提高性能。
摘要:
大视觉语言模型(LVLMs)已取得显著进展,但它们常常受到语言偏见的影响,产生答案时往往不依赖视觉证据。尽管先前工作试图通过解码策略、架构修改或精心挑选的指令数据来缓解这一问题,但它们通常缺乏对单个训练样本或token实际从图像中获益程度的定量衡量。在本工作中,我们引入了视觉信息增益(VIG),一种基于困惑度的度量指标,用于衡量视觉输入对预测不确定性的减少。VIG能够在样本和token层面进行细粒度分析,有效突出视觉基础元素,如颜色、空间关系和属性。借助这一指标,我们提出了一种VIG引导的定向训练方案,优先选择高VIG样本和token。这种方法提高了视觉基础性并减轻了语言偏见,通过专注于仅视觉信息丰富的样本和token,实现了显著减少监督下的优越性能。
英文摘要:
Large Vision Language Models (LVLMs) have achieved remarkable progress, yet they often suffer from language bias, producing answers without relying on visual evidence. While prior work attempts to mitigate this issue through decoding strategies, architectural modifications, or curated instruction data, they typically lack a quantitative measure of how much individual training samples or tokens actually benefit from the image. In this work, we introduce Visual Information Gain (VIG), a perplexity-based metric that measures the reduction in prediction uncertainty provided by visual input. VIG enables fine-grained analysis at both sample and token levels, effectively highlighting visually grounded elements such as colors, spatial relations, and attributes. Leveraging this, we propose a VIG-guided selective training scheme that prioritizes high-VIG samples and tokens. This approach improves visual grounding and mitigates language bias, achieving superior performance with significantly reduced supervision by focusing exclusively on visually informative samples and tokens.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。