arXiv 2507.195992025-07-25cs.CV

基于视觉定位与指代表达的以对象为中心的视频问答

Object-centric Video Question Answering with Visual Grounding and Referring

University of Amsterdam（阿姆斯特丹大学）；SAI, Shanghai Jiao Tong University（上海交通大学SAI研究所）；Xiaohongshu Inc（小红书公司）

Haochen Wang, Qirui Chen, Cilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, Weidi Xie, Stratis Gavves

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出支持视觉提示输入与对象定位输出的 VideoLLM，通过 STOM 传播跨帧视觉提示，并构建 VideoInfer 数据集，在12个基准上提升视频问答与分割性能。

摘要：

视频大语言模型（VideoLLM）近期在通用视频理解方面展现出显著进展。然而，现有模型主要关注高层理解，并且仅限于纯文本响应，这限制了以对象为中心的多轮交互的灵活性。在本文中，我们作出三项贡献：（i）我们通过引入一个 VideoLLM 模型来解决这些局限，该模型能够在视频推理任务中同时执行面向输入的对象指代表达和面向输出的定位，即允许用户使用文本和视觉提示与视频交互；（ii）我们提出 STOM（Spatial-Temporal Overlay Module，时空叠加模块），这是一种新颖方法，可将在任意单个时间戳输入的任意视觉提示传播到视频内的其余帧；（iii）我们提出 VideoInfer，这是一个人工整理的、以对象为中心的视频指令数据集，其特点是包含需要推理的问答对。我们在 VideoInfer 以及其他现有基准上，围绕视频问答和指代表象分割开展了全面实验。在 6 项任务的 12 个基准上的结果表明，我们提出的模型在视频问答和分割两方面均持续优于基线，凸显了其在多模态、以对象为中心的视频与图像理解中的鲁棒性。项目页面：https://qirui-chen.github.io/RGA3-release/。

英文摘要：

Video Large Language Models (VideoLLMs) have recently demonstrated remarkable progress in general video understanding. However, existing models primarily focus on high-level comprehension and are limited to text-only responses, restricting the flexibility for object-centric, multiround interactions. In this paper, we make three contributions: (i) we address these limitations by introducing a VideoLLM model, capable of performing both object referring for input and grounding for output in video reasoning tasks, i.e., allowing users to interact with videos using both textual and visual prompts; (ii) we propose STOM (Spatial-Temporal Overlay Module), a novel approach that propagates arbitrary visual prompts input at any single timestamp to the remaining frames within a video; (iii) we present VideoInfer, a manually curated object-centric video instruction dataset featuring questionanswering pairs that require reasoning. We conduct comprehensive experiments on VideoInfer and other existing benchmarks across video question answering and referring object segmentation. The results on 12 benchmarks of 6 tasks show that our proposed model consistently outperforms baselines in both video question answering and segmentation, underscoring its robustness in multimodal, object-centric video and image understanding. Project page: https://qirui-chen.github.io/RGA3-release/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。