iMOVE：实例运动感知的视频理解
iMOVE: Instance-Motion-Aware Video Understanding
AI总结：
针对视频大语言模型细粒度实例运动感知不足的问题，本文构建 iMOVE-IT 指令微调数据集，并提出 iMOVE 视频基础模型，以高效时空建模和相对位置令牌提升时序、通用及长时程视频理解。
摘要：
增强视频大语言模型细粒度实例时空运动感知能力，对于提升其时序理解和通用视频理解能力至关重要。然而，当前模型在感知详细且复杂的实例运动方面存在困难。为应对这些挑战，我们从数据和模型两个方面进行了改进。在数据方面，我们精心构建了 iMOVE-IT，这是首个大规模实例运动感知的视频指令微调数据集。该数据集包含全面的实例运动标注和时空互监督任务，为模型的实例运动感知提供了充分训练。在此基础上，我们提出 iMOVE，这是一种实例运动感知的视频基础模型，采用 Event-aware Spatiotemporal Efficient Modeling（事件感知时空高效建模），在保持计算效率的同时保留信息丰富的实例时空运动细节。它还引入 Relative Spatiotemporal Position Tokens（相对时空位置令牌），以确保对实例时空位置的感知。评估表明，iMOVE 不仅在视频时序理解和通用视频理解方面表现优异，还在长时程视频理解中展现出显著优势。
英文摘要：
Enhancing the fine-grained instance spatiotemporal motion perception capabilities of Video Large Language Models is crucial for improving their temporal and general video understanding. However, current models struggle to perceive detailed and complex instance motions. To address these challenges, we have made improvements from both data and model perspectives. In terms of data, we have meticulously curated iMOVE-IT, the first large-scale instance-motion-aware video instruction-tuning dataset. This dataset is enriched with comprehensive instance motion annotations and spatiotemporal mutual-supervision tasks, providing extensive training for the model's instance-motion-awareness. Building on this foundation, we introduce iMOVE, an instance-motion-aware video foundation model that utilizes Event-aware Spatiotemporal Efficient Modeling to retain informative instance spatiotemporal motion details while maintaining computational efficiency. It also incorporates Relative Spatiotemporal Position Tokens to ensure awareness of instance spatiotemporal positions. Evaluations indicate that iMOVE excels not only in video temporal understanding and general video understanding but also demonstrates significant advantages in long-term video understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
