从数据到建模：全开放词汇场景图生成
From Data to Modeling: Fully Open-vocabulary Scene Graph Generation
AI总结：
提出OvSGTR框架，利用类似DETR架构和关系感知预训练，实现全开放词汇场景图生成，在VG150上达到最先进性能。
摘要：
我们提出了OvSGTR，一种新颖的基于Transformer的框架，用于全开放词汇场景图生成，克服了传统封闭集模型的局限性。传统方法将对象和关系识别都限制在固定词汇表内，阻碍了它们在新概念频繁出现的现实世界场景中的适用性。相比之下，我们的方法联合预测超出预定义类别的对象（节点）及其相互关系（边）。OvSGTR利用类似DETR的架构，采用冻结的图像骨干网络和文本编码器来提取高质量的视觉和语义特征，然后通过Transformer解码器进行融合，实现端到端的场景图预测。为了丰富模型对复杂视觉关系的理解，我们提出了一种关系感知的预训练策略，以弱监督方式合成场景图标注。具体而言，我们研究了三种流程——基于场景解析器、基于LLM和基于多模态LLM——以最少的人工标注生成可迁移的监督信号。此外，我们通过引入视觉概念保持机制和知识蒸馏策略，解决了开放词汇设置中常见的灾难性遗忘问题，确保模型在微调期间保留丰富的语义线索。在VG150基准上的大量实验表明，OvSGTR在多种设置下均取得了最先进的性能，包括封闭集、基于开放词汇对象检测、基于关系以及全开放词汇场景。我们的结果凸显了大规模关系感知预训练和Transformer架构在推动场景图生成迈向更通用、更可靠的视觉理解方面的前景。
英文摘要：
We present OvSGTR, a novel transformer-based framework for fully open-vocabulary scene graph generation that overcomes the limitations of traditional closed-set models. Conventional methods restrict both object and relationship recognition to a fixed vocabulary, hindering their applicability to real-world scenarios where novel concepts frequently emerge. In contrast, our approach jointly predicts objects (nodes) and their inter-relationships (edges) beyond predefined categories. OvSGTR leverages a DETR-like architecture featuring a frozen image backbone and text encoder to extract high-quality visual and semantic features, which are then fused via a transformer decoder for end-to-end scene graph prediction. To enrich the model's understanding of complex visual relations, we propose a relation-aware pre-training strategy that synthesizes scene graph annotations in a weakly supervised manner. Specifically, we investigate three pipelines--scene parser-based, LLM-based, and multimodal LLM-based--to generate transferable supervision signals with minimal manual annotation. Furthermore, we address the common issue of catastrophic forgetting in open-vocabulary settings by incorporating a visual-concept retention mechanism coupled with a knowledge distillation strategy, ensuring that the model retains rich semantic cues during fine-tuning. Extensive experiments on the VG150 benchmark demonstrate that OvSGTR achieves state-of-the-art performance across multiple settings, including closed-set, open-vocabulary object detection-based, relation-based, and fully open-vocabulary scenarios. Our results highlight the promise of large-scale relation-aware pre-training and transformer architectures for advancing scene graph generation towards more generalized and reliable visual understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
