VideoRefer Suite：推进视频大语言模型中的时空对象理解
VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM
AI总结：
本文提出VideoRefer Suite，通过构建高质量对象级视频数据集、设计时空对象编码器模型及综合基准，显著提升视频大语言模型的细粒度时空对象理解与推理能力。
摘要：
视频大语言模型（Video LLMs）近期在通用视频理解方面展现出显著能力。然而，它们主要侧重于整体理解，难以捕捉细粒度的时空细节。此外，高质量的对象级视频指令数据及全面基准的缺乏进一步阻碍了其发展。为解决这些挑战，我们引入了VideoRefer Suite，以增强视频大语言模型进行更细粒度的时空视频理解，即实现对视频中任意对象的感知与推理。具体而言，我们从数据集、模型和基准三个方面全面开发了VideoRefer Suite。首先，我们引入一个多智能体数据引擎，精心构建了一个大规模、高质量的对象级视频指令数据集，名为VideoRefer-700K。其次，我们提出了VideoRefer模型，该模型配备了一个多功能的时空对象编码器，以捕获精确的区域和序列表示。最后，我们精心创建了VideoRefer-Bench，以全面评估视频大语言模型的时空理解能力，从多个方面进行评测。大量实验和分析表明，我们的VideoRefer模型不仅在视频指代基准上取得了令人满意的性能，还促进了通用视频理解能力的提升。
英文摘要：
Video Large Language Models (Video LLMs) have recently exhibited remarkable capabilities in general video understanding. However, they mainly focus on holistic comprehension and struggle with capturing fine-grained spatial and temporal details. Besides, the lack of high-quality object-level video instruction data and a comprehensive benchmark further hinders their advancements. To tackle these challenges, we introduce the VideoRefer Suite to empower Video LLM for finer-level spatial-temporal video understanding, i.e., enabling perception and reasoning on any objects throughout the video. Specially, we thoroughly develop VideoRefer Suite across three essential aspects: dataset, model, and benchmark. Firstly, we introduce a multi-agent data engine to meticulously curate a large-scale, high-quality object-level video instruction dataset, termed VideoRefer-700K. Next, we present the VideoRefer model, which equips a versatile spatial-temporal object encoder to capture precise regional and sequential representations. Finally, we meticulously create a VideoRefer-Bench to comprehensively assess the spatial-temporal understanding capability of a Video LLM, evaluating it across various aspects. Extensive experiments and analyses demonstrate that our VideoRefer model not only achieves promising performance on video referring benchmarks but also facilitates general video understanding capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
