SceneGraphVLM：基于视频的动态场景图生成方法
SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models
AI总结：
SceneGraphVLM通过紧凑的视觉语言模型生成视频场景图，采用TOON格式提升效率，结合监督微调与强化学习实现高精度与速度的平衡，适用于PSG、PVSG和Action Genome数据集。
摘要：
场景图生成为视觉感知提供紧凑的结构化表示，但从图像和视频中准确快速生成图仍具挑战性。最近基于VLM的方法可端到端生成结构化文本，但常产生冗长输出与无关对象和关系。我们提出SceneGraphVLM，一种紧凑的图像和视频场景图生成方法，使用小规模视觉语言模型。SceneGraphVLM以token高效的方式序列化图，并分两阶段训练模型：监督微调后接强化学习，使用hallucination-aware奖励平衡关系覆盖与精度，同时惩罚不支持的对象和关系。对于视频，模型可选择性地将每个帧条件在之前生成的图上，提供轻量级短期上下文，无需跟踪或后处理。我们评估SceneGraphVLM在PSG、PVSG和Action Genome上。通过紧凑的VLM和vLLM加速解码，SceneGraphVLM实现了强的质量-速度权衡，提升以精度为导向的SGG指标，同时保持合理的召回率，并在约1秒延迟内生成完整的场景图。代码和实现细节可在：https://github.com/markus0440/SceneGraphVLM.git获取。
英文摘要：
Scene graph generation provides a compact structured representation for visual perception, but accurate and fast graph prediction from images and videos remains challenging. Recent VLM-based methods can generate scene graphs end-to-end as structured text, yet often produce long outputs with irrelevant objects and relations. We present SceneGraphVLM, a compact method for image and video scene graph generation with small visual language models. SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages: supervised fine-tuning followed by reinforcement learning with hallucination-aware rewards that balance relation coverage and precision while penalizing unsupported objects and relations. For videos, the model can optionally condition each frame on the previously generated graph, providing lightweight short-term context without tracking or post-processing. We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome. With compact VLMs and vLLM-accelerated decoding, SceneGraphVLM achieves a strong quality-speed trade-off, improves precision-oriented SGG metrics while preserving reasonable recall, and generates complete scene graphs with approximately one-second latency. Code and implementation details are available at: https://github.com/markus0440/SceneGraphVLM.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
