超越生成：面向视觉语言模型事实准确性的多跳推理
Beyond Generation: Multi-Hop Reasoning for Factual Accuracy in Vision-Language Models
AI总结：
本文提出一种基于知识图谱的多跳推理框架，提升视觉语言模型在事实准确性上的表现，通过多步骤推理增强模型的逻辑推理能力。
摘要：
视觉语言模型（VLMs）是强大的生成工具，但常常由于缺乏稳健的推理能力而产生事实性不准确的输出。尽管在大型语言模型（LLMs）中已开展了大量关于整合外部知识进行推理的研究，但此类努力在VLMs中仍处于探索阶段，其中的挑战进一步加剧了多模态无缝衔接的需求。本文介绍了一种用于VLMs的知识引导推理框架，利用结构化知识图谱进行多跳验证，并通过图像-描述任务来展示我们的框架。我们的方法使能够系统地进行多步推理，包括视觉实体识别、知识图谱遍历以及基于事实的描述优化。我们通过分层、三元组基于和项目符号基于的知识表示来评估该框架，分析其在事实准确性及逻辑推理中的有效性。实验结果表明，我们的方法在初步实验中对Google Landmarks v2、Conceptual captions和Coco captions的混合数据集上，事实准确性提高了约31%，揭示了推理模式和失败模式的关键见解。本文展示了整合外部知识以推动VLMs推理能力的潜力，为更可靠和知识丰富的多模态系统铺平了道路。
英文摘要：
Visual Language Models (VLMs) are powerful generative tools but often produce factually inaccurate outputs due to a lack of robust reasoning capabilities. While extensive research has been conducted on integrating external knowledge for reasoning in large language models (LLMs), such efforts remain underexplored in VLMs, where the challenge is compounded by the need to bridge multiple modalities seamlessly. This work introduces a framework for knowledge-guided reasoning in VLMs, leveraging structured knowledge graphs for multi-hop verification using image-captioning task to illustrate our framework. Our approach enables systematic reasoning across multiple steps, including visual entity recognition, knowledge graph traversal, and fact-based caption refinement. We evaluate the framework using hierarchical, triple-based and bullet-point based knowledge representations, analyzing their effectiveness in factual accuracy and logical inference. Empirical results show that our approach improves factual accuracy by approximately 31% on preliminary experiments on a curated dataset of mixtures from Google Landmarks v2, Conceptual captions and Coco captions revealing key insights into reasoning patterns and failure modes. This work demonstrates the potential of integrating external knowledge for advancing reasoning in VLMs, paving the way for more reliable and knowledgable multimodal systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
