arXiv 2505.091182025-05-14cs.CV

看见场景之外：通过交互推理增强视觉-语言模型

Seeing Beyond the Scene: Enhancing Vision-Language Models with Interactional Reasoning

South China University of Technology（华南理工大学）；The Hong Kong Polytechnic University（香港理工大学）；Peng Cheng Laboratory（彭成实验室）

Dayong Liang, Changmeng Zheng, Zhiyuan Wen, Yi Cai, Xiao-Yong Wei, Qing Li

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对传统场景图偏重空间关系且难以泛化交互推理的问题，本文提出 ISGR 框架，通过双流图构建、交互查询和长期记忆强化学习增强 VLM，并在交互密集型基准上显著超越基线。

摘要：

传统场景图主要关注空间关系，限制了视觉-语言模型（VLM）对视觉场景中复杂交互进行推理的能力。本文解决两个关键挑战：（1）传统的检测到构建方法会产生不聚焦、上下文无关的关系集合；（2）现有方法无法形成持久记忆，以将交互推理泛化到新场景。我们提出 Interaction-augmented Scene Graph Reasoning（ISGR，交互增强场景图推理），这是一个通过三个互补组件增强 VLM 交互推理能力的框架。首先，我们的双流图构建器将基于 SAM 的空间关系提取与具备交互意识的字幕生成相结合，以生成具有空间 grounding 且功能显著的场景图。其次，我们采用有针对性的交互查询来激活 VLM 关于物体功能的潜在知识，将被动识别转化为对物体如何协同工作的主动推理。最后，我们引入一种长期记忆强化学习策略，配有专门面向交互的奖励函数，可将短暂模式转化为长期推理启发式规则。大量实验表明，我们的方法在交互密集型推理基准上显著优于基线方法，并在复杂场景理解任务上取得尤其强劲的提升。源代码可在 https://github.com/open_upon_acceptance 获取。

英文摘要：

Traditional scene graphs primarily focus on spatial relationships, limiting vision-language models' (VLMs) ability to reason about complex interactions in visual scenes. This paper addresses two key challenges: (1) conventional detection-to-construction methods produce unfocused, contextually irrelevant relationship sets, and (2) existing approaches fail to form persistent memories for generalizing interaction reasoning to new scenes. We propose Interaction-augmented Scene Graph Reasoning (ISGR), a framework that enhances VLMs' interactional reasoning through three complementary components. First, our dual-stream graph constructor combines SAM-powered spatial relation extraction with interaction-aware captioning to generate functionally salient scene graphs with spatial grounding. Second, we employ targeted interaction queries to activate VLMs' latent knowledge of object functionalities, converting passive recognition into active reasoning about how objects work together. Finally, we introduce a lone-term memory reinforcement learning strategy with a specialized interaction-focused reward function that transforms transient patterns into long-term reasoning heuristics. Extensive experiments demonstrate that our approach significantly outperforms baseline methods on interaction-heavy reasoning benchmarks, with particularly strong improvements on complex scene understanding tasks. The source code can be accessed at https://github.com/open_upon_acceptance.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。