可解释的零样本指代表达理解与查询驱动的场景图
Interpretable Zero-shot Referring Expression Comprehension with Query-driven Scene Graphs
AI总结：
本文提出SGREC方法，通过查询驱动的场景图作为结构化中介，结合视觉语言模型和大语言模型，实现可解释的零样本指代表达理解，在多个基准测试中取得优异成绩。
摘要：
零样本指代表达理解（REC）旨在在没有任务特定训练数据的情况下，通过自然语言查询定位图像中的目标对象，需要强大的视觉理解能力。现有视觉语言模型（VLMs），如CLIP，通常通过直接测量文本查询和图像区域之间的特征相似性来解决零样本REC问题。然而，这些方法难以捕捉细粒度的视觉细节和理解复杂的对象关系。同时，大语言模型（LLMs）在高层语义推理方面表现出色，但无法直接将视觉特征抽象为文本语义，限制了其在REC任务中的应用。为克服这些限制，我们提出了SGREC，一种利用查询驱动的场景图作为结构化中介的可解释零样本REC方法。具体来说，我们首先使用VLM构建查询驱动的场景图，该图明确编码了空间关系、描述性标题和与给定查询相关的对象交互。通过利用此场景图，我们弥合了低级图像区域和LLMs所需高层语义理解之间的差距。最后，LLM从场景图提供的结构化文本表示中推断出目标对象，并通过详细的解释说明其决策，确保推理过程的可解释性。广泛的实验表明，SGREC在大多数零样本REC基准测试中实现了最高准确率，包括RefCOCO val（66.78%）、RefCOCO+ testB（53.43%）和RefCOCOg val（73.28%），突显了其强大的视觉场景理解能力。
英文摘要：
Zero-shot referring expression comprehension (REC) aims to locate target objects in images given natural language queries without relying on task-specific training data, demanding strong visual understanding capabilities. Existing Vision-Language Models~(VLMs), such as CLIP, commonly address zero-shot REC by directly measuring feature similarities between textual queries and image regions. However, these methods struggle to capture fine-grained visual details and understand complex object relationships. Meanwhile, Large Language Models~(LLMs) excel at high-level semantic reasoning, their inability to directly abstract visual features into textual semantics limits their application in REC tasks. To overcome these limitations, we propose \textbf{SGREC}, an interpretable zero-shot REC method leveraging query-driven scene graphs as structured intermediaries. Specifically, we first employ a VLM to construct a query-driven scene graph that explicitly encodes spatial relationships, descriptive captions, and object interactions relevant to the given query. By leveraging this scene graph, we bridge the gap between low-level image regions and higher-level semantic understanding required by LLMs. Finally, an LLM infers the target object from the structured textual representation provided by the scene graph, responding with detailed explanations for its decisions that ensure interpretability in the inference process. Extensive experiments show that SGREC achieves top-1 accuracy on most zero-shot REC benchmarks, including RefCOCO val (66.78\%), RefCOCO+ testB (53.43\%), and RefCOCOg val (73.28\%), highlighting its strong visual scene understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
