运动基础视频推理：在像素级理解与感知运动
Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level
AI总结：
本文提出了运动基础视频推理任务及大规模数据集GROUNDMORE，要求模型根据问题生成视频分割掩码以实现隐式时空推理，并引入结合多模态LLM、SAM和轻量级定位头的基线模型MORA，在性能上平均提升21.5%。
摘要：
在本文中，我们介绍了运动基础视频推理，这是一种新的运动理解任务，需要根据输入问题生成视觉答案（视频分割掩码），因此需要隐式的时空推理和基础定位。该任务将专注于显式动作/运动基础定位的现有时空基础工作，通过问题实现隐式推理，扩展为更通用的格式。为了促进新任务的发展，我们收集了一个名为 GROUNDMORE 的大规模数据集，包含 1,715 个视频片段和 249K 个对象掩码，这些掩码经过精心设计，包含 4 种问题类型（因果、顺序、反事实和描述性），用于对深度且全面的运动推理能力进行基准测试。GROUNDMORE 独特地要求模型生成视觉答案，与纯文本相比，提供了更具体且视觉上可解释的响应。它在时空基础定位和推理两方面评估模型，旨在应对运动相关视频推理、时间感知和像素级理解中的复杂挑战。此外，我们引入了一个名为运动基础视频推理助手（MORA）的新型基线模型。MORA 结合了多模态大语言模型（Multimodal LLM）的多模态推理能力、基础模型（SAM）的像素级感知能力，以及轻量级定位头的时间感知能力。MORA 在 GROUNDMORE 上取得了令人满意的性能，相对最佳现有视觉基础基线模型平均提升了 21.5%。我们希望这一新颖且具挑战性的任务，能为通过视频推理分割实现鲁棒且通用的运动理解的未来进展铺平道路。
英文摘要：
In this paper, we introduce Motion-Grounded Video Reasoning, a new motion understanding task that requires generating visual answers (video segmentation masks) according to the input question, and hence needs implicit spatiotemporal reasoning and grounding. This task extends existing spatiotemporal grounding work focusing on explicit action/motion grounding, to a more general format by enabling implicit reasoning via questions. To facilitate the development of the new task, we collect a large-scale dataset called GROUNDMORE, which comprises 1,715 video clips, 249K object masks that are deliberately designed with 4 question types (Causal, Sequential, Counterfactual, and Descriptive) for benchmarking deep and comprehensive motion reasoning abilities. GROUNDMORE uniquely requires models to generate visual answers, providing a more concrete and visually interpretable response than plain texts. It evaluates models on both spatiotemporal grounding and reasoning, fostering to address complex challenges in motion-related video reasoning, temporal perception, and pixel-level understanding. Furthermore, we introduce a novel baseline model named Motion-Grounded Video Reasoning Assistant (MORA). MORA incorporates the multimodal reasoning ability from the Multimodal LLM, the pixel-level perception capability from the grounding model (SAM), and the temporal perception ability from a lightweight localization head. MORA achieves respectable performance on GROUNDMORE outperforming the best existing visual grounding baseline model by an average of 21.5% relatively. We hope this novel and challenging task will pave the way for future advancements in robust and general motion understanding via video reasoning segmentation
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
