ViGoR：通过细粒度奖励建模提升大视觉语言模型的视觉 grounding
ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling
AI总结：
ViGoR通过细粒度奖励建模提升大视觉语言模型的视觉 grounding 能力，采用更经济的人类评估和自动化方法，有效提高视觉推理准确性。
摘要：
通过将自然语言理解、生成能力和广泛知识与图像感知相结合，最近的大视觉语言模型（LVLMs）展示了前所未有的视觉推理能力。然而，生成的文本常常在视觉输入中存在不准确的 grounding，导致错误，例如虚构不存在的场景元素、遗漏场景的重要部分，以及推断对象的错误属性和关系。为了解决这些问题，我们引入了一个新的框架，ViGoR（通过细粒度奖励建模实现视觉 grounding），该框架利用细粒度奖励建模显著提升 LVLMs 在预训练基线上的视觉 grounding 能力。这种改进通过更经济的人类评估而不是完整的监督学习以及自动化方法高效实现。我们通过各种评估方法和基准测试展示了我们方法的有效性。此外，我们发布了我们的注释（https://github.com/amazon-science/vigor），包含15,440张图像和生成文本对，进行了细粒度评估，以促进社区的相关研究。
英文摘要：
By combining natural language understanding, generation capabilities, and breadth of knowledge of large language models with image perception, recent large vision language models (LVLMs) have shown unprecedented visual reasoning capabilities. However, the generated text often suffers from inaccurate grounding in the visual input, resulting in errors such as hallucination of nonexistent scene elements, missing significant parts of the scene, and inferring incorrect attributes of and relationships between objects. To address these issues, we introduce a novel framework, ViGoR (Visual Grounding Through Fine-Grained Reward Modeling) that utilizes fine-grained reward modeling to significantly enhance the visual grounding of LVLMs over pre-trained baselines. This improvement is efficiently achieved using much cheaper human evaluations instead of full supervisions, as well as automated methods. We show the effectiveness of our approach through a variety of evaluation methods and benchmarks. Additionally, we released our human annotation (https://github.com/amazon-science/vigor) comprising 15,440 images and generated text pairs with fine-grained evaluations to contribute to related research in the community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
