基于二级跟踪和强化学习验证的多模态大模型高效时空定位
Efficient Spatio-Temporal Grounding with Multimodal Large Models via Second-Level Tracking and RL Verification
AI总结：
提出一种从帧级到秒级跟踪的流水线，结合跨秒平滑、链式思维轨迹合成和强化学习优化，在长视频中实现高效且准确的时空定位。
摘要：
长视频中的时空定位需要精确的时间定位和基于自然语言查询的鲁棒目标跟踪。虽然最近的视觉语言模型（VLM）展现出强大的推理能力，但将逐帧推理直接应用于长序列在计算上昂贵且不稳定。我们提出了一种实用的流水线，将帧级跟踪转变为秒级跟踪，并执行跨秒平滑以保持连续性同时减少序列长度。为了改进推理监督，我们使用先进的多模态模型合成链式思维风格的轨迹，用于时间定位和目标选择，并用真实标注替换生成的时空坐标以避免噪声监督。我们进一步使用基于 $t\_\mathrm{IoU}+mv\_\mathrm{IoU}$ 的验证器通过强化学习优化策略。在多个FPS设置下的实验表明，我们的方法在效率和定位质量之间实现了良好的平衡。
英文摘要：
Spatio-temporal grounding in long videos requires precise temporal localization and robust object tracking conditioned on natural-language queries. While recent vision-language models (VLMs) show strong reasoning ability, directly applying frame-by-frame inference to long sequences is computationally expensive and unstable. We propose a practical pipeline that shifts from frame-level to second-level tracking and performs cross-second smoothing to preserve continuity while reducing sequence length. To improve reasoning supervision, we synthesize chain-of-thought style trajectories using advanced multimodal models for temporal localization and target selection, and replace generated spatio-temporal coordinates with ground-truth annotations to avoid noisy supervision. We further optimize the policy with reinforcement learning using a verifier based on $t\_\mathrm{IoU}+mv\_\mathrm{IoU}$. Experiments across multiple FPS settings show that our method achieves a strong trade-off between efficiency and localization quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
