SPARROW：在像素级视频MLLM中学习空间精度和时间参照一致性
SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs
AI总结：
SPARROW通过引入目标特定跟踪特征和双提示设计，提升视频像素级理解的空间精度和时间一致性，基于30646个视频和45231对问答对的数据集，实现六个基准测试的显著改进。
摘要：
多模态大语言模型（MLLMs）已从图像级推理发展到像素级定位，但将其能力扩展到视频仍具挑战性，因为模型必须实现空间精度和时间一致的参考跟踪。现有视频MLLMs通常依赖静态分割标记（[SEG]）进行帧级定位，提供语义但缺乏时间上下文，导致物体移动或重新出现时出现空间漂移、身份切换和不稳定初始化。我们引入SPARROW，一种像素级视频MLLM，通过两个关键组件统一空间精度和时间稳定性：（i）目标特定跟踪特征（TSF），在训练期间注入时间对齐的参照线索；（ii）双提示设计，解码框（[BOX]）和分割（[SEG]）标记以融合几何先验与语义定位。SPARROW由一个精心编排的参照视频数据集支持，包含30646个视频和45231对问答对，并通过类无关的SAM2基提出器实现端到端操作。整合到三个最近的开源视频MLLMs（UniPixel、GLUS和VideoGLaMM）中，SPARROW在六个基准测试中实现一致的提升，改进高达+8.9 J&F在RVOS、+5 mIoU在视觉定位和+5.4 CLAIR在GCG。这些结果表明，SPARROW显著提高了像素级视频理解中的参照稳定性、空间精度和时间一致性。项目页面：https://risys-lab.github.io/SPARROW
英文摘要：
Multimodal large language models (MLLMs) have advanced from image-level reasoning to pixel-level grounding, but extending these capabilities to videos remains challenging as models must achieve spatial precision and temporally consistent reference tracking. Existing video MLLMs often rely on a static segmentation token ([SEG]) for frame-wise grounding, which provides semantics but lacks temporal context, causing spatial drift, identity switches, and unstable initialization when objects move or reappear. We introduce SPARROW, a pixel-grounded video MLLM that unifies spatial accuracy and temporal stability through two key components: (i) Target-Specific Tracked Features (TSF), which inject temporally aligned referent cues during training, and (ii) a dual-prompt design that decodes box ([BOX]) and segmentation ([SEG]) tokens to fuse geometric priors with semantic grounding. SPARROW is supported by a curated referential video dataset of 30,646 videos and 45,231 Q&A pairs and operates end-to-end without external detectors via a class-agnostic SAM2-based proposer. Integrated into three recent open-source video MLLMs (UniPixel, GLUS, and VideoGLaMM), SPARROW delivers consistent gains across six benchmarks, improving up to +8.9 J&F on RVOS, +5 mIoU on visual grounding, and +5.4 CLAIR on GCG. These results demonstrate that SPARROW substantially improves referential stability, spatial precision, and temporal coherence in pixel-grounded video understanding. Project page: https://risys-lab.github.io/SPARROW
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
