arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.21778cs.CV

Scene-VLM:通过视觉-语言模型进行多模态视频场景分割

Scene-VLM: Multimodal Video Scene Segmentation via Vision-Language Models

Ben-Gurion University(本·古里安大学);Amazon Prime Video(亚马逊Prime视频);Tel-Aviv University(特拉维夫大学)

Nimrod Berman, Adam Botach, Emanuel Ben-Baruch, Shunit Haviv Hakimi, Asaf Gendler, Ilan Naiman, Erez Yosef, Igor Kviatkovsky

AI总结:

本文提出Scene-VLM,首个基于视觉-语言模型的视频场景分割框架,通过融合视觉与文本信息实现多模态推理,提升场景分割的准确性和可解释性。

摘要:

对长视频进行语义连贯的场景分割是大规模视频理解中的基础任务。现有基于编码器的方法受限于视觉偏见,无法利用序列依赖性,且缺乏叙事理解和可解释性。本文提出Scene-VLM,首个经过微调的视觉-语言模型(VLM)框架,用于视频场景分割。Scene-VLM联合处理视觉和文本线索,包括帧、转录和可选元数据,以实现跨连续镜头的多模态推理。模型按顺序生成预测,具有镜头间的因果依赖性,并引入上下文聚焦窗口机制,确保每个镜头级决策有足够的时间上下文。此外,我们提出了一种从VLM的token级logits中提取置信度分数的方案,使精度-召回率的可控制权衡成为可能,此前仅限于基于编码器的方法。进一步,我们证明,通过最小的针对性监督,我们的模型可以对齐生成连贯的自然语言理由,以解释其边界决策。我们的方法在标准场景分割基准上实现了最先进的性能。例如,在MovieNet上,Scene-VLM相比先前领先方法在AP上提高了6,在F1上提高了13.7。

英文摘要:

Segmenting long-form videos into semantically coherent scenes is a fundamental task in large-scale video understanding. Existing encoder-based methods are limited by visual-centric biases, classify each shot in isolation without leveraging sequential dependencies, and lack both narrative understanding and explainability. In this paper, we present Scene-VLM, the first fine-tuned vision-language model (VLM) framework for video scene segmentation. Scene-VLM jointly processes visual and textual cues including frames, transcriptions, and optional metadata to enable multimodal reasoning across consecutive shots. The model generates predictions sequentially with causal dependencies among shots and introduces a context-focus window mechanism to ensure sufficient temporal context for each shot-level decision. In addition, we propose a scheme to extract confidence scores from the token-level logits of the VLM, enabling controllable precision-recall trade-offs that were previously limited to encoder-based methods. Furthermore, we demonstrate that our model can be aligned to generate coherent natural-language rationales for its boundary decisions through minimal targeted supervision. Our approach achieves state-of-the-art performance on standard scene segmentation benchmarks. On MovieNet, for example, Scene-VLM yields significant improvements of +6 AP and +13.7 F1 over the previous leading method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑