arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.02309cs.CVcs.MM
收藏

TR-DETR:用于联合时刻检索与高光检测的任务互惠Transformer

TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection

Central China Normal University(华中师范大学);Hubei University of Technology(湖北工业大学)

Hao Sun, Mingyao Zhou, Wenjing Chen, Wei Xie

AI总结:

本文提出TR-DETR模型,通过局部-全局多模态对齐、视觉特征细化和任务协作模块探索视频时刻检索与高光检测间的互惠性,在多个数据集上优于现有最先进方法。

摘要:

基于自然语言查询的视频时刻检索(MR)和高光检测(HD)是两个高度相关的任务,旨在获取视频内的相关时刻以及每个视频片段的高光分数。近期,一些方法致力于构建基于DETR的网络来联合解决MR和HD。这些方法在多模态特征提取和特征交互后简单添加两个独立的任务头,取得了良好性能。然而,这些方法未充分利用两个任务间的互惠关系。本文提出基于DETR的任务互惠Transformer(TR-DETR),专注于探索MR与HD间的固有互惠性。具体而言,首先构建局部-全局多模态对齐模块,将来自不同模态的特征对齐到共享潜在空间。随后,设计视觉特征细化模块,以消除视觉特征中与查询无关的信息,用于模态交互。最后,构建任务协作模块,利用MR与HD间的互惠性来细化检索流程和高光分数预测过程。在QVHighlights、Charades-STA和TVSum数据集上的综合实验表明,TR-DETR优于现有最先进方法。代码可在https://github.com/mingyao1120/TR-DETR获取。

英文摘要:

Video moment retrieval (MR) and highlight detection (HD) based on natural language queries are two highly related tasks, which aim to obtain relevant moments within videos and highlight scores of each video clip. Recently, several methods have been devoted to building DETR-based networks to solve both MR and HD jointly. These methods simply add two separate task heads after multi-modal feature extraction and feature interaction, achieving good performance. Nevertheless, these approaches underutilize the reciprocal relationship between two tasks. In this paper, we propose a task-reciprocal transformer based on DETR (TR-DETR) that focuses on exploring the inherent reciprocity between MR and HD. Specifically, a local-global multi-modal alignment module is first built to align features from diverse modalities into a shared latent space. Subsequently, a visual feature refinement is designed to eliminate query-irrelevant information from visual features for modal interaction. Finally, a task cooperation module is constructed to refine the retrieval pipeline and the highlight score prediction process by utilizing the reciprocity between MR and HD. Comprehensive experiments on QVHighlights, Charades-STA and TVSum datasets demonstrate that TR-DETR outperforms existing state-of-the-art methods. Codes are available at \url{https://github.com/mingyao1120/TR-DETR}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑