Elysium：通过MLLM探索视频中的对象级感知
Elysium: Exploring Object-level Perception in Videos via MLLM
AI总结：
本文提出端到端可训练的MLLM方法Elysium及大规模数据集ElysiumTrack-1M和令牌压缩模型T-Selector，以解决视频对象级感知中的预训练数据不足与计算负担问题。
摘要：
多模态大语言模型（MLLMs）已经展示了其在静态图像中感知对象的能力，但它们在视频相关任务（如对象跟踪）中的应用仍未得到充分研究。这种探索的缺乏主要源于两个关键挑战。首先，需要在大规模视频数据集上进行广泛预训练，以使MLLMs具备跨多帧感知对象并理解帧间关系的能力。其次，在大语言模型（LLMs）的上下文窗口内处理大量帧会带来显著的计算负担。为了解决第一个挑战，我们引入了ElysiumTrack-1M，这是一个大规模视频数据集，支持三项任务：单对象跟踪（SOT）、指称单对象跟踪（RSOT）和视频指称表达生成（Video-REG）。ElysiumTrack-1M包含127万个带标注的视频帧，并配有相应的对象框和描述。利用该数据集，我们对MLLMs进行了训练，并提出了一个令牌压缩模型T-Selector以应对第二个挑战。我们提出的方法Elysium：通过MLLM探索视频中的对象级感知，是一个端到端可训练的MLLM，旨在无需任何额外插件或专家模型即可在视频中执行对象级任务。所有代码和数据集均可在https://github.com/Hon-Wong/Elysium获取。
英文摘要：
Multi-modal Large Language Models (MLLMs) have demonstrated their ability to perceive objects in still images, but their application in video-related tasks, such as object tracking, remains understudied. This lack of exploration is primarily due to two key challenges. Firstly, extensive pretraining on large-scale video datasets is required to equip MLLMs with the capability to perceive objects across multiple frames and understand inter-frame relationships. Secondly, processing a large number of frames within the context window of Large Language Models (LLMs) can impose a significant computational burden. To address the first challenge, we introduce ElysiumTrack-1M, a large-scale video dataset supported for three tasks: Single Object Tracking (SOT), Referring Single Object Tracking (RSOT), and Video Referring Expression Generation (Video-REG). ElysiumTrack-1M contains 1.27 million annotated video frames with corresponding object boxes and descriptions. Leveraging this dataset, we conduct training of MLLMs and propose a token-compression model T-Selector to tackle the second challenge. Our proposed approach, Elysium: Exploring Object-level Perception in Videos via MLLM, is an end-to-end trainable MLLM that attempts to conduct object-level tasks in videos without requiring any additional plug-in or expert models. All codes and datasets are available at https://github.com/Hon-Wong/Elysium.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
