多时相指代分割的开源基准与基线
An Open-Source Benchmark and Baseline for Multi-temporal Referring Segmentation
AI总结：
提出多时相指代分割任务，通过自动化数据构建管道CRAFT-Agent生成首个基准MTRefSeg-21K，并设计两阶段训练的变化感知LVLM框架MTRefSeg-R1，实现优于现有基线的性能。
摘要：
大型视觉语言模型（LVLMs）展现了强大的视觉理解和语言引导定位能力，但其多时相视觉推理能力仍未充分探索。为填补这一空白，我们引入了 extbf{多时相指代分割（MTRS）}，这是一个新任务，旨在从多时相图像中分割语言描述的时间变化。MTRS通过联合要求时相对应推理、语言定位和像素级掩码预测，扩展了传统的指代分割和变化检测。我们提出了 extbf{CRAFT-Agent}，一个带有人工审核的自动化数据构建管道，并构建了 extbf{MTRefSeg-21K}，这是第一个MTRS基准，包含21K个高质量的多时相图像-文本-掩码三元组，覆盖多样化的场景、视角和领域。对一系列基于VLM和LVLM的模型进行基准测试表明，直接推理表现较差，而任务特定的微调仍然有限。为解决这一问题，我们提出了 extbf{MTRefSeg-R1}，一个采用两阶段策略训练的变化感知LVLM框架。它首先从20K个仅视觉的双时相样本中学习通用时间变化感知，然后在MTRefSeg-21K上进行微调，以实现细粒度的语言引导时间定位。MTRefSeg-R1显式建模跨时相视觉差异，将语言指令与时间变化对齐，并预测所指变化掩码。大量实验表明，与现有的LVLM基线相比，MTRefSeg-R1实现了强大且通常更优的性能，展示了MTRS的挑战和潜力。
英文摘要：
Large Vision-Language Models (LVLMs) have shown strong visual understanding and language-guided grounding abilities, yet their capacity for multi-temporal visual reasoning remains underexplored. To bridge this gap, we introduce \textbf{Multi-temporal Referring Segmentation (MTRS)}, a new task that aims to segment language-described temporal changes from multi-temporal images. MTRS extends conventional referring segmentation and change detection by jointly requiring temporal correspondence reasoning, language grounding, and pixel-level mask prediction. We propose \textbf{CRAFT-Agent}, an automated data construction pipeline with human auditing, and build \textbf{MTRefSeg-21K}, the first MTRS benchmark, containing 21K high-quality multi-temporal image-text-mask triplets across diverse scenes, viewpoints, and domains. Benchmarking a broad set of VLM- and LVLM-based models reveals that direct inference performs poorly, while task-specific fine-tuning remains limited. To address this, we propose \textbf{MTRefSeg-R1}, a change-aware LVLM framework trained with a two-stage strategy. It first learns general temporal-change perception from 20K vision-only bi-temporal samples, and is then fine-tuned on MTRefSeg-21K for fine-grained language-guided temporal localization. MTRefSeg-R1 explicitly models cross-temporal visual differences, aligns language instructions with temporal variations, and predicts referred change masks. Extensive experiments show that MTRefSeg-R1 achieves strong and often superior performance compared with existing LVLM baselines, demonstrating the challenge and potential of MTRS.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
