arXiv 2512.055242025-12-08cs.CV

VOST-SGG：基于视觉语言模型的一阶段时空场景图生成

VOST-SGG: VLM-Aided One-Stage Spatio-Temporal Scene Graph Generation

College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算与数据科学学院）；Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore（科学、技术与研究局高性能计算研究所）；Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore（科学、技术与研究局前沿人工智能研究中心）

Chinthani Sugandhika, Chen Li, Deepu Rajan, Basura Fernando

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

VOST-SGG通过整合视觉语言模型的常识推理能力，提出了一种基于多模态特征和双源查询初始化的一阶段时空场景图生成方法，实现了对视频中对象及其关系的高效建模。

摘要：

时空场景图生成（ST-SGG）旨在建模视频帧中对象及其随时间变化的关系，从而为下游推理任务如视频描述和视觉问答提供可解释的表示。尽管最近在DETR风格的一阶段ST-SGG模型上取得了进展，但它们仍然存在几个关键限制。首先，虽然这些模型依赖于基于注意力的可学习查询作为核心组件，但这些可学习查询在语义上无信息且实例无关地初始化。其次，这些模型仅依赖单模态视觉特征进行谓词分类。为了解决这些挑战，我们提出了VOST-SGG，一种基于视觉语言模型（VLMs）的一阶段ST-SGG框架，将VLMs的常识推理能力整合到ST-SGG流程中。首先，我们引入了双源查询初始化策略，将关注什么与关注哪里解耦，从而实现语义 grounded 的what-where推理。此外，我们提出了一种多模态特征库，融合来自VLMs的视觉、文本和空间线索，以改进谓词分类。在Action Genome数据集上的大量实验表明，我们的方法取得了最先进的性能，验证了将VLM辅助的语义先验和多模态特征整合到ST-SGG中的有效性。我们将在https://github.com/LUNAProject22/VOST上发布代码。

英文摘要：

Spatio-temporal scene graph generation (ST-SGG) aims to model objects and their evolving relationships across video frames, enabling interpretable representations for downstream reasoning tasks such as video captioning and visual question answering. Despite recent advancements in DETR-style single-stage ST-SGG models, they still suffer from several key limitations. First, while these models rely on attention-based learnable queries as a core component, these learnable queries are semantically uninformed and instance-agnostically initialized. Second, these models rely exclusively on unimodal visual features for predicate classification. To address these challenges, we propose VOST-SGG, a VLM-aided one-stage ST-SGG framework that integrates the common sense reasoning capabilities of vision-language models (VLMs) into the ST-SGG pipeline. First, we introduce the dual-source query initialization strategy that disentangles what to attend to from where to attend, enabling semantically grounded what-where reasoning. Furthermore, we propose a multi-modal feature bank that fuses visual, textual, and spatial cues derived from VLMs for improved predicate classification. Extensive experiments on the Action Genome dataset demonstrate that our approach achieves state-of-the-art performance, validating the effectiveness of integrating VLM-aided semantic priors and multi-modal features for ST-SGG. We will release the code at https://github.com/LUNAProject22/VOST.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。