使用场景图将结构化表示引入预训练视觉与语言模型
Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs
AI总结：
研究提出通过场景图（SGs）将结构化信息整合到预训练视觉与语言模型（VLMs）的视觉和文本表示中，以增强组合场景理解，同时保持零样本能力。
摘要：
视觉与语言模型（VLMs）在多种任务中展现了出色的零样本（ZS）性能。然而，近期研究表明，即使是最好的VLMs也难以捕捉组合场景理解的各个方面，如物体属性、关系和动作状态。相比之下，获取能够改进这些模型的结构化标注（如场景图，SGs）既耗时又昂贵，因此无法大规模使用。在此，我们探究小型SG数据集是否能提供充足信息，以增强预训练VLMs的结构化理解能力。我们证明，通过将包含结构化信息的组件整合到视觉和文本表示中，确实能够在从SGs学习时改进VLMs。在视觉端，我们在图像transformer中加入一个特殊的“SG组件”，训练其预测SG信息；在文本端，我们利用SGs生成细粒度描述，突出场景的不同组合方面。我们的方法在多个VL数据集上提升了多种流行VLMs的性能，且仅导致ZS能力的轻微下降。
英文摘要：
Vision and language models (VLMs) have demonstrated remarkable zero-shot (ZS) performance in a variety of tasks. However, recent works have shown that even the best VLMs struggle to capture aspects of compositional scene understanding, such as object attributes, relations, and action states. In contrast, obtaining structured annotations, such as scene graphs (SGs), that could improve these models is time-consuming and costly, and thus cannot be used on a large scale. Here we ask whether small SG datasets can provide sufficient information for enhancing structured understanding of pretrained VLMs. We show that it is indeed possible to improve VLMs when learning from SGs by integrating components that incorporate structured information into both visual and textual representations. For the visual side, we incorporate a special "SG Component" in the image transformer trained to predict SG information, while for the textual side, we utilize SGs to generate fine-grained captions that highlight different compositional aspects of the scene. Our method improves the performance of several popular VLMs on multiple VL datasets with only a mild degradation in ZS capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
