arXiv 2311.109882024-10-07cs.CV

拓展场景图边界：通过视觉-概念对齐与保留实现完全开放词汇场景图生成

Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention

The Hong Kong Polytechnic University（香港理工大学）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（香港中文大学（深圳）人工智能与机器人中心（HKISI-CAS））；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）

Zuyao Chen, Jinlin Wu, Zhen Lei, Zhaoxiang Zhang, Changwen Chen

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对传统场景图生成的闭集局限，研究提出统一框架OvSGTR，通过视觉-概念对齐、关系感知预训练及知识蒸馏，实现完全开放词汇的场景图生成，在Visual Genome基准上验证了有效性。

摘要：

场景图生成（Scene Graph Generation, SGG）提供了一种结构化表示，在众多计算机视觉应用中至关重要。然而，传统SGG方法受限于闭集假设，仅能识别预定义的对象和关系类别。为克服这一局限，我们基于节点和边将SGG场景划分为四种不同设置：闭集SGG、基于开放词汇（对象）检测的SGG（OvD-SGG）、基于开放词汇关系的SGG（OvR-SGG）以及基于开放词汇检测+关系的SGG（OvD+R-SGG）。尽管以对象为中心的开放词汇SGG近年来已有研究，但涉及关系的更具挑战性的开放词汇SGG问题仍相对未被探索。为填补这一空白，我们从整体视角提出了一个名为OvSGTR的统一框架，旨在实现完全开放词汇SGG。该框架是一种端到端Transformer架构，可同时为节点和边学习视觉-概念对齐，使模型能够识别未见过的类别。针对涉及关系的开放词汇SGG这一更具挑战性的设置，所提方法整合了利用图像-字幕数据的关系感知预训练，并通过知识蒸馏保留视觉-概念对齐。在Visual Genome基准上的综合实验结果证明了所提框架的有效性和优越性。我们的代码可在https://github.com/gpt4vision/OvSGTR/ 获取。

英文摘要：

Scene Graph Generation (SGG) offers a structured representation critical in many computer vision applications. Traditional SGG approaches, however, are limited by a closed-set assumption, restricting their ability to recognize only predefined object and relation categories. To overcome this, we categorize SGG scenarios into four distinct settings based on the node and edge: Closed-set SGG, Open Vocabulary (object) Detection-based SGG (OvD-SGG), Open Vocabulary Relation-based SGG (OvR-SGG), and Open Vocabulary Detection + Relationbased SGG (OvD+R-SGG). While object-centric open vocabulary SGG has been studied recently, the more challenging problem of relation-involved open-vocabulary SGG remains relatively unexplored. To fill this gap, we propose a unified framework named OvSGTR towards fully open vocabulary SGG from a holistic view. The proposed framework is an end-to-end transformer architecture, which learns a visual-concept alignment for both nodes and edges, enabling the model to recognize unseen categories. For the more challenging settings of relation-involved open vocabulary SGG, the proposed approach integrates relation-aware pretraining utilizing image-caption data and retains visual-concept alignment through knowledge distillation. Comprehensive experimental results on the Visual Genome benchmark demonstrate the effectiveness and superiority of the proposed framework. Our code is available at https://github.com/gpt4vision/OvSGTR/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。