从像素到图：基于视觉语言模型的开放词汇场景图生成
From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models
AI总结：
针对现有方法难以生成新颖视觉关系概念场景图的问题，提出一种基于序列生成的开放词汇SGG框架，利用VLM进行图像到文本生成并构建场景图，不仅提升了开放词汇SGG性能，还增强了下游视觉语言任务。
摘要：
场景图生成（SGG）旨在将视觉场景解析为用于下游推理任务的中间图表示。尽管近期取得了进展，但现有方法难以生成具有新颖视觉关系概念的场景图。为应对这一挑战，我们引入了一种基于序列生成的全新开放词汇SGG框架。该框架通过结合图像到图的生成范式来利用视觉语言预训练模型（VLM）。具体而言，我们利用VLM通过图像到文本生成来产生场景图序列，随后从这些序列构建场景图。通过这种方式，我们利用了VLM在开放词汇SGG方面的强大能力，并无缝集成了显式关系建模以增强VL任务。实验结果表明，我们的设计不仅在开放词汇下实现了卓越性能，还通过显式关系建模知识提升了下游视觉语言任务的性能。
英文摘要：
Scene graph generation (SGG) aims to parse a visual scene into an intermediate graph representation for downstream reasoning tasks. Despite recent advancements, existing methods struggle to generate scene graphs with novel visual relation concepts. To address this challenge, we introduce a new open-vocabulary SGG framework based on sequence generation. Our framework leverages vision-language pre-trained models (VLM) by incorporating an image-to-graph generation paradigm. Specifically, we generate scene graph sequences via image-to-text generation with VLM and then construct scene graphs from these sequences. By doing so, we harness the strong capabilities of VLM for open-vocabulary SGG and seamlessly integrate explicit relational modeling for enhancing the VL tasks. Experimental results demonstrate that our design not only achieves superior performance with an open vocabulary but also enhances downstream vision-language task performance through explicit relation modeling knowledge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
