层次化局部-全局Transformer用于时间语句定位
Hierarchical Local-Global Transformer for Temporal Sentence Grounding
AI总结：
提出层次化局部-全局Transformer（HLGT），通过建模视频和查询的不同粒度层次及跨模态交互，实现更细粒度的多模态表示，并在三个数据集上取得最先进性能。
摘要：
本文研究多媒体问题中的时间语句定位（TSG），旨在根据给定的句子查询准确确定未修剪视频中的特定视频片段。传统的TSG方法主要遵循自上而下或自下而上的框架，且不是端到端的，严重依赖耗时的后处理来优化定位结果。最近，一些基于Transformer的方法被提出，以高效有效地建模视频和查询之间的细粒度语义对齐。尽管这些方法在一定程度上取得了显著性能，但它们将视频帧和查询词等同视为Transformer输入进行关联，未能捕捉它们不同粒度的不同语义。为解决这一问题，本文提出了一种新颖的层次化局部-全局Transformer（HLGT），利用这种层次信息并建模不同粒度层次和不同模态之间的交互，以学习更细粒度的多模态表示。具体来说，我们首先将视频和查询分割成单独的片段和短语，通过时间Transformer学习它们的局部上下文（相邻依赖）和全局相关性（长距离依赖）。然后，引入全局-局部Transformer来学习局部级和全局级语义之间的交互，以实现更好的多模态推理。此外，我们开发了一种新的跨模态循环一致性损失，以强制两个模态之间的交互并鼓励它们之间的语义对齐。最后，我们设计了一种全新的跨模态并行Transformer解码器，用于整合编码的视觉和文本特征以进行最终定位。在三个具有挑战性的数据集上进行的大量实验表明，我们提出的HLGT实现了新的最先进性能。
英文摘要：
This paper studies the multimedia problem of temporal sentence grounding (TSG), which aims to accurately determine the specific video segment in an untrimmed video according to a given sentence query. Traditional TSG methods mainly follow the top-down or bottom-up framework and are not end-to-end. They severely rely on time-consuming post-processing to refine the grounding results. Recently, some transformer-based approaches are proposed to efficiently and effectively model the fine-grained semantic alignment between video and query. Although these methods achieve significant performance to some extent, they equally take frames of the video and words of the query as transformer input for correlating, failing to capture their different levels of granularity with distinct semantics. To address this issue, in this paper, we propose a novel Hierarchical Local-Global Transformer (HLGT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities for learning more fine-grained multi-modal representations. Specifically, we first split the video and query into individual clips and phrases to learn their local context (adjacent dependency) and global correlation (long-range dependency) via a temporal transformer. Then, a global-local transformer is introduced to learn the interactions between the local-level and global-level semantics for better multi-modal reasoning. Besides, we develop a new cross-modal cycle-consistency loss to enforce interaction between two modalities and encourage the semantic alignment between them. Finally, we design a brand-new cross-modal parallel transformer decoder to integrate the encoded visual and textual features for final grounding. Extensive experiments on three challenging datasets show that our proposed HLGT achieves a new state-of-the-art performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
