arXiv 2507.057982025-07-08cs.CV

SPADE：面向开放词汇全景场景图生成的空间感知去噪网络，具备长程与局部上下文推理

SPADE: Spatial-Aware Denoising Network for Open-vocabulary Panoptic Scene Graph Generation with Long- and Local-range Context Reasoning

The Laboratory of Intelligent Collaborative Computing of UESTC（UESTC智能协同计算实验室）；Ubiquitous Intelligence and Trusted Services Key Laboratory of Sichuan Province（四川省 Ubiquitous Intelligence and Trusted Services 重点实验室）；Monash University（墨尔本大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））

Xin Hu, Ke Qin, Guiduo Duan, Ming Li, Yuan-Fang Li, Tao He

备注：: Accepted to ICCV 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对开放词汇全景场景图生成中VLM空间推理不足的问题，提出SPADE框架，利用扩散模型反转校准和空间感知图变换器，在PSG和Visual Genome数据集上超越现有方法。

摘要：

全景场景图生成（PSG）将实例分割与关系理解相结合，以捕捉复杂场景中像素级的结构关系。尽管近期利用预训练视觉-语言模型（VLM）的方法在开放词汇设置下显著提升了性能，但它们通常忽略了VLM在空间关系推理中的固有限制，例如难以区分对象的相对位置，这导致关系预测次优。受去噪扩散模型在保留输入图像空间结构方面的反转过程的启发，我们提出了SPADE（空间感知去噪网络）框架——一种用于开放词汇PSG的新方法。SPADE包含两个关键步骤：（1）用于UNet适配的反转引导校准，以及（2）空间感知上下文推理。在第一步中，我们通过一种基于轻量级LoRA的微调策略，利用反转过程中获得的交叉注意力图，将通用的预训练教师扩散模型校准为PSG专用的去噪网络。在第二步中，我们开发了一个空间感知关系图变换器，该变换器同时捕获局部和长程上下文信息，有助于生成高质量的关系查询。在基准PSG和Visual Genome数据集上的大量实验表明，SPADE在封闭集和开放集场景中均优于最先进的方法，尤其是在空间关系预测方面。

英文摘要：

Panoptic Scene Graph Generation (PSG) integrates instance segmentation with relation understanding to capture pixel-level structural relationships in complex scenes. Although recent approaches leveraging pre-trained vision-language models (VLMs) have significantly improved performance in the open-vocabulary setting, they commonly ignore the inherent limitations of VLMs in spatial relation reasoning, such as difficulty in distinguishing object relative positions, which results in suboptimal relation prediction. Motivated by the denoising diffusion model's inversion process in preserving the spatial structure of input images, we propose SPADE (SPatial-Aware Denoising-nEtwork) framework -- a novel approach for open-vocabulary PSG. SPADE consists of two key steps: (1) inversion-guided calibration for the UNet adaptation, and (2) spatial-aware context reasoning. In the first step, we calibrate a general pre-trained teacher diffusion model into a PSG-specific denoising network with cross-attention maps derived during inversion through a lightweight LoRA-based fine-tuning strategy. In the second step, we develop a spatial-aware relation graph transformer that captures both local and long-range contextual information, facilitating the generation of high-quality relation queries. Extensive experiments on benchmark PSG and Visual Genome datasets demonstrate that SPADE outperforms state-of-the-art methods in both closed- and open-set scenarios, particularly for spatial relationship prediction.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文