从长期含噪视频中学习多粒度对应关系
Multi-granularity Correspondence Learning from Long-term Noisy Videos
AI总结：
针对长视频语言学习中存在的多粒度噪声对应问题，提出基于最优传输的Norton框架，从粗细两粒度解决错位问题，在多项视频任务上验证了有效性。
摘要：
现有视频语言研究主要聚焦于短视频片段的学习，由于长视频建模的计算成本过高，长期时间依赖关系鲜有被探究。为解决这一问题，学习视频片段与字幕之间的对应关系是一种可行方案，但该方法不可避免地会遇到多粒度噪声对应（Multi-granularity Noisy Correspondence，MNC）问题。具体而言，MNC指的是片段-字幕错位（粗粒度）和帧-单词错位（细粒度），会阻碍时间学习与视频理解。本文提出噪声鲁棒时间最优传输（NOise Robust Temporal Optimal traNsport，Norton）方法，在统一的最优传输（optimal transport，OT）框架下解决MNC问题。简言之，Norton基于最优传输，采用视频-段落对比损失与片段-字幕对比损失来捕捉长期依赖关系。为解决视频-段落对比中的粗粒度错位问题，Norton通过可对齐提示桶过滤掉不相关的片段与字幕，并基于传输距离重新对齐异步的片段-字幕对。为解决细粒度错位问题，Norton引入软最大化算子来识别关键单词与关键帧。此外，Norton通过最优传输分配校正对齐目标，利用片段-字幕对比中潜在的错误负样本，以确保精准的时间建模。在视频检索、视频问答（videoQA）和动作分割任务上的大量实验验证了该方法的有效性。代码可在https://lin-yijie.github.io/projects/Norton获取。
英文摘要：
Existing video-language studies mainly focus on learning short video clips, leaving long-term temporal dependencies rarely explored due to over-high computational cost of modeling long videos. To address this issue, one feasible solution is learning the correspondence between video clips and captions, which however inevitably encounters the multi-granularity noisy correspondence (MNC) problem. To be specific, MNC refers to the clip-caption misalignment (coarse-grained) and frame-word misalignment (fine-grained), hindering temporal learning and video understanding. In this paper, we propose NOise Robust Temporal Optimal traNsport (Norton) that addresses MNC in a unified optimal transport (OT) framework. In brief, Norton employs video-paragraph and clip-caption contrastive losses to capture long-term dependencies based on OT. To address coarse-grained misalignment in video-paragraph contrast, Norton filters out the irrelevant clips and captions through an alignable prompt bucket and realigns asynchronous clip-caption pairs based on transport distance. To address the fine-grained misalignment, Norton incorporates a soft-maximum operator to identify crucial words and key frames. Additionally, Norton exploits the potential faulty negative samples in clip-caption contrast by rectifying the alignment target with OT assignment to ensure precise temporal modeling. Extensive experiments on video retrieval, videoQA, and action segmentation verify the effectiveness of our method. Code is available at https://lin-yijie.github.io/projects/Norton.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
