Omni-RGPT：通过Token Mark统一图像与视频的区域级理解
Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks
AI总结：
本研究提出多模态大模型Omni-RGPT，通过Token Mark机制统一图像与视频的区域级理解，构建RegVID-300k数据集，在多项视觉语言任务上达到SOTA。
摘要：
我们提出Omni-RGPT，这是一款多模态大语言模型，旨在实现图像与视频的区域级理解。为在时空维度上获得一致的区域表示，我们引入Token Mark（令牌标记），即一组在视觉特征空间中高亮目标区域的令牌。这些令牌通过区域提示（如边界框或掩码）直接嵌入空间区域，同时被加入文本提示以指定目标，在视觉令牌与文本令牌间建立直接关联。为在无需轨迹的前提下进一步实现鲁棒的视频理解，我们提出一项辅助任务，利用令牌的一致性引导Token Mark，实现视频中稳定的区域解读。此外，我们构建了一个大规模区域级视频指令数据集RegVID-300k。Omni-RGPT在基于图像和视频的常识推理基准上取得了当前最优结果，同时在字幕生成和指代表达理解任务中表现优异。
英文摘要：
We present Omni-RGPT, a multimodal large language model designed to facilitate region-level comprehension for both images and videos. To achieve consistent region representation across spatio-temporal dimensions, we introduce Token Mark, a set of tokens highlighting the target regions within the visual feature space. These tokens are directly embedded into spatial regions using region prompts (e.g., boxes or masks) and simultaneously incorporated into the text prompt to specify the target, establishing a direct connection between visual and text tokens. To further support robust video understanding without requiring tracklets, we introduce an auxiliary task that guides Token Mark by leveraging the consistency of the tokens, enabling stable region interpretation across the video. Additionally, we introduce a large-scale region-level video instruction dataset (RegVID-300k). Omni-RGPT achieves state-of-the-art results on image and video-based commonsense reasoning benchmarks while showing strong performance in captioning and referring expression comprehension tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
