跨多域标签的混合学习视频时刻检索
Hybrid-Learning Video Moment Retrieval across Multi-Domain Labels
AI总结：
提出混合学习视频时刻检索方法和多分支视频-文本对齐模型EVA，通过跨域知识迁移改善弱标签目标域的检索性能。
摘要：
视频时刻检索（VMR）是指根据给定的文本查询描述（句子），在未剪辑的原始视频中搜索视觉时间时刻。现有研究要么从收集目标时刻时间边界上详尽的逐帧标注开始（全监督），要么仅使用视频级别的视频-文本配对标签进行学习（弱监督）。前者由于在昂贵的标注成本下数据集规模和多样性受限，对未知概念和/或新场景的泛化能力较差；后者则因标签不完整而受到视觉-文本错误相关的影响。在这项工作中，我们引入了一种称为混合学习视频时刻检索的新方法，通过将全监督源域学习到的视频-文本匹配关系适应到弱标签目标域来解决该问题，当它们不共享公共标签空间时进行知识迁移。我们的目标是探索两个域之间的共享通用知识，以改善弱标签目标域中的模型学习。具体而言，我们引入了一个多分支视频-文本对齐模型（EVA），该模型执行跨模态（视觉-文本）匹配信息共享和多模态特征对齐，以优化域不变的视觉和文本特征以及每个任务具有判别性的联合视频-文本表示。实验表明，EVA在探索源域中的时间段标注以帮助在目标域中学习无时间标签的视频时刻检索方面是有效的。
英文摘要：
Video moment retrieval (VMR) is to search for a visual temporal moment in an untrimmed raw video by a given text query description (sentence). Existing studies either start from collecting exhaustive frame-wise annotations on the temporal boundary of target moments (fully-supervised), or learn with only the video-level video-text pairing labels (weakly-supervised). The former is poor in generalisation to unknown concepts and/or novel scenes due to restricted dataset scale and diversity under expensive annotation costs; the latter is subject to visual-textual mis-correlations from incomplete labels. In this work, we introduce a new approach called hybrid-learning video moment retrieval to solve the problem by knowledge transfer through adapting the video-text matching relationships learned from a fully-supervised source domain to a weakly-labelled target domain when they do not share a common label space. Our aim is to explore shared universal knowledge between the two domains in order to improve model learning in the weakly-labelled target domain. Specifically, we introduce a multiplE branch Video-text Alignment model (EVA) that performs cross-modal (visual-textual) matching information sharing and multi-modal feature alignment to optimise domain-invariant visual and textual features as well as per-task discriminative joint video-text representations. Experiments show EVA's effectiveness in exploring temporal segment annotations in a source domain to help learn video moment retrieval without temporal labels in a target domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
