EtC: 面向多模态大语言模型弱监督视频定位的时序边界先扩展后厘清方法
EtC: Temporal Boundary Expand then Clarify for Weakly Supervised Video Grounding with Multimodal Large Language Model
AI总结：
针对弱监督视频定位中伪边界质量差的问题，提出EtC框架，借助MLLM扩展初始伪边界并结合互学习与定制对比目标厘清噪声，在两个数据集上取得更优性能。
摘要：
早期弱监督视频定位（WSVG）方法因缺少时序边界标注，常面临边界检测不完整的问题。为填补视频级与边界级标注之间的差距，显式监督方法（即生成伪时序边界用于训练）已取得显著成效。但这类方法中的数据增强操作可能会破坏关键时序信息，导致生成的伪边界质量较差。\n本文提出一种全新思路，在保留原始时序内容完整性的同时，引入更有价值的信息来扩展不完整的边界。基于此我们提出EtC（Expand then Clarify，先扩展后厘清）框架：首先利用额外信息扩展初始的不完整伪边界，随后对扩展后的边界进行细化以得到精准边界。受视频连续性（即相邻帧之间存在视觉相似性）的启发，我们使用强大的多模态大语言模型（MLLMs）对初始伪边界内的每一帧进行标注，为扩展后的边界生成更全面的描述。\n为进一步厘清扩展边界中的噪声，我们将互学习与定制化的候选框级对比目标相结合，通过可学习的方式在“不完整但干净的（初始）边界”与“全面但含噪的（扩展）边界”之间取得平衡，从而得到更精准的边界。实验结果表明，我们的方法在两个具有挑战性的WSVG数据集上表现出优越性能。
英文摘要：
Early weakly supervised video grounding (WSVG) methods often struggle with incomplete boundary detection due to the absence of temporal boundary annotations. To bridge the gap between video-level and boundary-level annotation, explicit-supervision methods, i.e., generating pseudo-temporal boundaries for training, have achieved great success. However, data augmentations in these methods might disrupt critical temporal information, yielding poor pseudo boundaries. In this paper, we propose a new perspective that maintains the integrity of the original temporal content while introducing more valuable information for expanding the incomplete boundaries. To this end, we propose EtC (Expand then Clarify), first use the additional information to expand the initial incomplete pseudo boundaries, and subsequently refine these expanded ones to achieve precise boundaries. Motivated by video continuity, i.e., visual similarity across adjacent frames, we use powerful multimodal large language models (MLLMs) to annotate each frame within initial pseudo boundaries, yielding more comprehensive descriptions for expanded boundaries. To further clarify the noise of expanded boundaries, we combine mutual learning with a tailored proposal-level contrastive objective to use a learnable approach to harmonize a balance between incomplete yet clean (initial) and comprehensive yet noisy (expanded) boundaries for more precise ones. Experiments demonstrate the superiority of our method on two challenging WSVG datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
