SAMA：面向大语言模型的多轮指代定位视频对话
SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models
AI总结：
针对视频大模型时空理解不足的问题，本文提出SAMA模型、SAMA-239K数据集和SAMA-Bench基准，实现多轮指代定位视频对话，并刷新通用定位基准性能。
摘要：
在视频中实现细粒度的时空理解仍然是当前视频大型多模态模型（Video LMMs）面临的主要挑战。解决这一挑战需要掌握两个核心能力：视频指代理解（video referring understanding），即捕捉视频区域的语义；以及视频定位（video grounding），即基于自然语言描述分割对象区域。然而，现有的大多数方法将这些任务孤立地处理，限制了朝着统一、指代定位的视频交互的进展。我们识别出一个关键瓶颈，即缺乏高质量、统一的视频指令数据以及用于评估指代定位视频对话的全面基准。为了解决这些挑战，我们从三个核心方面做出贡献：数据集、模型和基准。首先，我们引入了SAMA-239K，这是一个大规模数据集，包含15K个视频，专门策划以支持视频指代理解、定位和多轮视频对话的联合学习。其次，我们提出了SAMA模型，该模型整合了一个多功能的时空上下文聚合器和一个Segment Anything Model，以联合增强细粒度视频理解和精确的定位能力。最后，我们建立了SAMA-Bench，一个精心设计的基准，包含来自522个视频的5,067个问题，以全面评估Video LMMs在多轮、时空指代理解和定位对话中的综合能力。大量的实验和基准测试结果表明，SAMA不仅在SAMA-Bench上取得了强劲的性能，而且在通用定位基准上树立了新的最先进水平，同时在标准视觉理解基准上保持了极具竞争力的性能。
英文摘要：
Achieving fine-grained spatio-temporal understanding in videos remains a major challenge for current Video Large Multimodal Models (Video LMMs). Addressing this challenge requires mastering two core capabilities: video referring understanding, which captures the semantics of video regions, and video grounding, which segments object regions based on natural language descriptions. However, most existing approaches tackle these tasks in isolation, limiting progress toward unified, referentially grounded video interaction. We identify a key bottleneck in the lack of high-quality, unified video instruction data and a comprehensive benchmark for evaluating referentially grounded video chat. To address these challenges, we contribute in three core aspects: dataset, model, and benchmark. First, we introduce SAMA-239K, a large-scale dataset comprising 15K videos specifically curated to enable joint learning of video referring understanding, grounding, and multi-turn video chat. Second, we propose the SAMA model, which incorporates a versatile spatio-temporal context aggregator and a Segment Anything Model to jointly enhance fine-grained video comprehension and precise grounding capabilities. Finally, we establish SAMA-Bench, a meticulously designed benchmark consisting of 5,067 questions from 522 videos, to comprehensively evaluate the integrated capabilities of Video LMMs in multi-turn, spatio-temporal referring understanding and grounded dialogue. Extensive experiments and benchmarking results show that SAMA not only achieves strong performance on SAMA-Bench but also sets a new state-of-the-art on general grounding benchmarks, while maintaining highly competitive performance on standard visual understanding benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
