利用检索增强标签提升复杂场景下的大规模视觉语言理解
Leveraging Retrieval-Augmented Tags for Large Vision-Language Understanding in Complex Scenes
AI总结：
提出VRAP框架，通过检索增强对象标签增强大型视觉语言模型，在复杂场景中实现细粒度对象感知推理，并在多个基准上达到最先进性能，同时降低40%推理延迟。
摘要：
在视觉语言任务中，对象感知推理对当前模型提出了重大挑战，尤其是在处理未见对象、减少幻觉以及捕捉复杂视觉场景中的细粒度关系方面。为解决这些局限性，我们提出了视觉感知检索增强提示（Vision-Aware Retrieval-Augmented Prompting, VRAP）框架，这是一种生成式方法，通过将检索增强的对象标签整合到提示中，增强大规模视觉语言模型（LVLMs）的能力。VRAP引入了一种新颖的流程，其中结构化标签（包括对象、属性和关系）通过预训练的视觉编码器和场景图解析器提取。这些标签通过外部知识进行丰富，并纳入大语言模型（LLM）的输入，从而实现详细且准确的推理。我们在多个视觉语言基准上评估了VRAP，包括VQAv2、GQA、VizWiz和COCO，在细粒度推理和多模态理解方面取得了最先进的性能。此外，我们的消融研究强调了检索增强标签和对比学习的重要性，而人工评估则确认了VRAP生成准确、详细且上下文相关响应的能力。值得注意的是，VRAP通过消除运行时检索，将推理延迟降低了40%。这些结果表明，VRAP是推进对象感知多模态推理的稳健且高效的框架。
英文摘要：
Object-aware reasoning in vision-language tasks poses significant challenges for current models, particularly in handling unseen objects, reducing hallucinations, and capturing fine-grained relationships in complex visual scenes. To address these limitations, we propose the Vision-Aware Retrieval-Augmented Prompting (VRAP) framework, a generative approach that enhances Large Vision-Language Models (LVLMs) by integrating retrieval-augmented object tags into their prompts. VRAP introduces a novel pipeline where structured tags, including objects, attributes, and relationships, are extracted using pretrained visual encoders and scene graph parsers. These tags are enriched with external knowledge and incorporated into the LLM's input, enabling detailed and accurate reasoning. We evaluate VRAP across multiple vision-language benchmarks, including VQAv2, GQA, VizWiz, and COCO, achieving state-of-the-art performance in fine-grained reasoning and multimodal understanding. Additionally, our ablation studies highlight the importance of retrieval-augmented tags and contrastive learning, while human evaluations confirm VRAP's ability to generate accurate, detailed, and contextually relevant responses. Notably, VRAP achieves a 40% reduction in inference latency by eliminating runtime retrieval. These results demonstrate that VRAP is a robust and efficient framework for advancing object-aware multimodal reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
