arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.00355cs.CVcs.AI
收藏

MH-DETR:基于跨模态Transformer的视频片段与高光检测

MH-DETR: Video Moment and Highlight Detection with Cross-modal Transformer

Nanjing University(南京大学);Hubei Normal University(湖北师范大学);MedAI Technology (Wuxi) Co., Ltd.(无锡医渡云科技有限公司);Technical University of Munich(慕尼黑工业大学)

Yifang Xu, Yunzhuo Sun, Yang Li, Yilei Shi, Xiaoxiang Zhu, Sidan Du

AI总结:

针对现有基于DETR的视频片段与高光检测方法存在的模态融合粗糙、跨模态交互不足问题,提出MH-DETR,通过单模态池化算子和跨模态交互模块提升性能,在多个数据集上超越现有SOTA方法。

摘要:

随着视频理解需求的不断增长,视频片段与高光检测(MHD)已成为一项重要的研究课题。MHD旨在同时定位所有片段并预测逐片段的显著性分数。尽管现有基于DETR的方法已取得进展,但我们发现这些方法对不同模态的特征进行粗粒度融合,这会削弱时序模态内上下文,导致跨模态交互不充分。为解决该问题,我们提出了专为MHD定制的MH-DETR(片段与高光检测Transformer)。具体而言,我们在单模态编码器中引入了一种简单且高效的池化算子,以捕获全局模态内上下文。此外,为了获得时序对齐的跨模态特征,我们在编码器与解码器之间设计了一个即插即用的跨模态交互模块,可无缝整合视觉与文本特征。在QVHighlights、Charades-STA、Activity-Net和TVSum数据集上的综合实验表明,MH-DETR的性能优于现有最先进方法,证明了其有效性与优越性。我们的代码可在https://github.com/YoucanBaby/MH-DETR获取。

英文摘要:

With the increasing demand for video understanding, video moment and highlight detection (MHD) has emerged as a critical research topic. MHD aims to localize all moments and predict clip-wise saliency scores simultaneously. Despite progress made by existing DETR-based methods, we observe that these methods coarsely fuse features from different modalities, which weakens the temporal intra-modal context and results in insufficient cross-modal interaction. To address this issue, we propose MH-DETR (Moment and Highlight Detection Transformer) tailored for MHD. Specifically, we introduce a simple yet efficient pooling operator within the uni-modal encoder to capture global intra-modal context. Moreover, to obtain temporally aligned cross-modal features, we design a plug-and-play cross-modal interaction module between the encoder and decoder, seamlessly integrating visual and textual features. Comprehensive experiments on QVHighlights, Charades-STA, Activity-Net, and TVSum datasets show that MH-DETR outperforms existing state-of-the-art methods, demonstrating its effectiveness and superiority. Our code is available at https://github.com/YoucanBaby/MH-DETR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑