多对时序句子定位的多线程知识迁移网络
Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network
AI总结：
提出多对时序句子定位新任务，并设计多线程知识迁移网络，通过跨模态对比、原型对齐和自适应负样本选择实现多对视频-查询对的协同训练。
摘要：
给定一些包含未修剪视频和句子查询的视频-查询对，时序句子定位（TSG）旨在定位这些视频中与查询相关的片段。尽管先前优秀的TSG方法取得了显著成功，但它们单独训练每个视频-查询对，忽略了不同对之间的关系。我们观察到，相似的视频/查询内容不仅有助于TSG模型更好地理解和泛化跨模态表示，还能帮助模型定位一些复杂的视频-查询对。先前的方法遵循单线程框架，无法共同训练不同的对，并且通常花费大量时间重新获取冗余知识，限制了其实际应用。为此，在本文中，我们提出了一种全新的设置：多对TSG，旨在共同训练这些对。特别地，我们提出了一种新颖的视频-查询共同训练方法，即多线程知识迁移网络，以有效且高效地定位各种视频-查询对。首先，我们挖掘不同查询之间的空间和时间语义以相互协作。为了同时学习模态内和模态间表示，我们设计了一个跨模态对比模块，通过自监督策略探索语义一致性。为了充分对齐不同对之间的视觉和文本表示，我们设计了一种原型对齐策略，以1）匹配对象原型和短语原型以实现空间对齐，以及2）对齐活动原型和句子原型以实现时间对齐。最后，我们开发了一个自适应负样本选择模块，以自适应地生成跨模态匹配的阈值。大量实验表明了我们提出方法的有效性和效率。
英文摘要：
Given some video-query pairs with untrimmed videos and sentence queries, temporal sentence grounding (TSG) aims to locate query-relevant segments in these videos. Although previous respectable TSG methods have achieved remarkable success, they train each video-query pair separately and ignore the relationship between different pairs. We observe that the similar video/query content not only helps the TSG model better understand and generalize the cross-modal representation but also assists the model in locating some complex video-query pairs. Previous methods follow a single-thread framework that cannot co-train different pairs and usually spends much time re-obtaining redundant knowledge, limiting their real-world applications. To this end, in this paper, we pose a brand-new setting: Multi-Pair TSG, which aims to co-train these pairs. In particular, we propose a novel video-query co-training approach, Multi-Thread Knowledge Transfer Network, to locate a variety of video-query pairs effectively and efficiently. Firstly, we mine the spatial and temporal semantics across different queries to cooperate with each other. To learn intra- and inter-modal representations simultaneously, we design a cross-modal contrast module to explore the semantic consistency by a self-supervised strategy. To fully align visual and textual representations between different pairs, we design a prototype alignment strategy to 1) match object prototypes and phrase prototypes for spatial alignment, and 2) align activity prototypes and sentence prototypes for temporal alignment. Finally, we develop an adaptive negative selection module to adaptively generate a threshold for cross-modal matching. Extensive experiments show the effectiveness and efficiency of our proposed method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
