arXiv 2512.109452025-12-11cs.CV

MeViS：一种多模态数据集，用于指称运动表达视频分割

MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation

Fudan University（复旦大学）；Shanghai University of Finance and Economics（上海财经大学）；Nanyang Technological University（南洋理工大学）

Henghui Ding, Chang Liu, Shuting He, Kaining Ying, Xudong Jiang, Chen Change Loy, Yu-Gang Jiang

备注：: IEEE TPAMI, Project Page: https://henghuiding.com/MeViS/
发表信息：: in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 12, pp. 11400-11416, 2025
DOI：: 10.1109/TPAMI.2025.3600507

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

MeViS数据集通过多模态数据提升视频分割中运动表达的理解能力，提出LMPM++方法实现新突破。

摘要：

本文提出了一种大规模的多模态数据集，用于指称运动表达视频分割，重点在于基于物体运动的语言描述来分割和跟踪目标对象。现有的指称视频分割数据集往往侧重于显著对象，并使用富含静态属性的语言表达，可能允许在单帧中识别目标对象。此类数据集在视频和语言中低估了运动的作用。为了探索使用运动表达和运动推理线索进行像素级视频理解的可行性，我们引入了MeViS，该数据集包含33,072个由人类标注的运动表达，涵盖8,171个对象在2,006个复杂场景视频中的文本和音频。我们对15种现有方法在MeViS支持的4个任务上进行了基准测试，包括6种指称视频对象分割（RVOS）方法、3种音频引导视频对象分割（AVOS）方法、2种指称多对象跟踪（RMOT）方法以及4种视频描述方法，用于新引入的指称运动表达生成（RMEG）任务。结果表明现有方法在处理运动表达引导的视频理解方面存在弱点和局限性。我们进一步分析了挑战，并提出了一种LMPM++方法用于RVOS/AVOS/RMOT，实现了新的最先进结果。我们的数据集提供了一个平台，促进了复杂视频场景中运动表达引导的视频理解算法的发展。所提出的MeViS数据集和方法的源代码已公开在https://henghuiding.com/MeViS/。

英文摘要：

This paper proposes a large-scale multi-modal dataset for referring motion expression video segmentation, focusing on segmenting and tracking target objects in videos based on language description of objects' motions. Existing referring video segmentation datasets often focus on salient objects and use language expressions rich in static attributes, potentially allowing the target object to be identified in a single frame. Such datasets underemphasize the role of motion in both videos and languages. To explore the feasibility of using motion expressions and motion reasoning clues for pixel-level video understanding, we introduce MeViS, a dataset containing 33,072 human-annotated motion expressions in both text and audio, covering 8,171 objects in 2,006 videos of complex scenarios. We benchmark 15 existing methods across 4 tasks supported by MeViS, including 6 referring video object segmentation (RVOS) methods, 3 audio-guided video object segmentation (AVOS) methods, 2 referring multi-object tracking (RMOT) methods, and 4 video captioning methods for the newly introduced referring motion expression generation (RMEG) task. The results demonstrate weaknesses and limitations of existing methods in addressing motion expression-guided video understanding. We further analyze the challenges and propose an approach LMPM++ for RVOS/AVOS/RMOT that achieves new state-of-the-art results. Our dataset provides a platform that facilitates the development of motion expression-guided video understanding algorithms in complex video scenes. The proposed MeViS dataset and the method's source code are publicly available at https://henghuiding.com/MeViS/

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。