VideoMolmo：时空定位与指向的结合
VideoMolmo: Spatio-Temporal Grounding Meets Pointing
AI总结：
VideoMolmo 提出一种基于 Molmo 架构的大型多模态模型，通过时间注意力模块和 SAM2 双向点传播实现细粒度时空指向，并构建 72k 视频数据集及 VPoS-Bench 基准，显著提升时空定位精度与推理能力。
摘要：
时空定位对于从生物研究到自主导航和交互式界面等各个领域的精确交互至关重要。当前基于视频的方法虽然在跟踪方面表现出色，但缺乏大型语言模型的复杂推理能力，限制了它们的上下文理解和泛化能力。我们提出了 VideoMolmo，这是一种大型多模态模型，专为基于文本描述的细粒度时空指向而定制。VideoMolmo 建立在 Molmo 架构之上，引入了一个时间模块，利用注意力机制使每一帧以先前帧为条件，确保时间一致性。此外，我们新颖的时间掩码融合流程采用 SAM2 进行双向点传播，显著增强了视频序列中的连贯性。这种两步分解，即首先使用 LLM 生成精确的指向坐标，然后依赖顺序掩码融合模块生成连贯的分割，不仅简化了语言模型的任务，还增强了解释性。由于缺乏合适的数据集，我们构建了一个包含 72k 个视频-字幕对并标注了 100k 个对象点的综合数据集。为了评估 VideoMolmo 的泛化能力，我们引入了 VPoS-Bench，这是一个具有挑战性的分布外基准，涵盖五个真实世界场景：细胞跟踪、以自我为中心的视觉、自动驾驶、视频-GUI 交互和机器人技术。我们还在指称视频对象分割（Refer-VOS）和推理 VOS 任务上评估了我们的模型。与现有模型相比，VideoMolmo 显著提高了时空指向精度和推理能力。我们的代码和模型已在 https://github.com/mbzuai-oryx/VideoMolmo 公开。
英文摘要：
Spatio-temporal localization is vital for precise interactions across diverse domains, from biological research to autonomous navigation and interactive interfaces. Current video-based approaches, while proficient in tracking, lack the sophisticated reasoning capabilities of large language models, limiting their contextual understanding and generalization. We introduce VideoMolmo, a large multimodal model tailored for fine-grained spatio-temporal pointing conditioned on textual descriptions. Building upon the Molmo architecture, VideoMolmo incorporates a temporal module utilizing an attention mechanism to condition each frame on preceding frames, ensuring temporal consistency. Additionally, our novel temporal mask fusion pipeline employs SAM2 for bidirectional point propagation, significantly enhancing coherence across video sequences. This two-step decomposition, i.e., first using the LLM to generate precise pointing coordinates, then relying on a sequential mask-fusion module to produce coherent segmentation, not only simplifies the task for the language model but also enhances interpretability. Due to the lack of suitable datasets, we curate a comprehensive dataset comprising 72k video-caption pairs annotated with 100k object points. To evaluate the generalization of VideoMolmo, we introduce VPoS-Bench, a challenging out-of-distribution benchmark spanning five real-world scenarios: Cell Tracking, Egocentric Vision, Autonomous Driving, Video-GUI Interaction, and Robotics. We also evaluate our model on Referring Video Object Segmentation (Refer-VOS) and Reasoning VOS tasks. In comparison to existing models, VideoMolmo substantially improves spatio-temporal pointing accuracy and reasoning capability. Our code and models are publicly available at https://github.com/mbzuai-oryx/VideoMolmo.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
