LD-DETR：用于视频时刻检索和精彩片段检测的循环解码器检测变换器
LD-DETR: Loop Decoder DEtection TRansformer for Video Moment Retrieval and Highlight Detection
AI总结：
针对视频时刻检索和精彩片段检测中语义重叠、局部特征提取不足及解码不充分的问题，提出LD-DETR模型，通过相似度矩阵蒸馏、卷积局部特征提取和循环解码器，在多个基准上达到最优性能。
摘要：
视频时刻检索和精彩片段检测旨在基于文本查询在视频中找到相应内容。现有模型通常首先使用对比学习方法对齐视频和文本特征，然后融合并提取多模态信息，最后使用Transformer解码器对多模态信息进行解码。然而，现有方法面临几个问题：（1）数据集中不同样本之间的重叠语义信息阻碍了模型的多模态对齐性能；（2）现有模型无法高效提取视频的局部特征；（3）现有模型使用的Transformer解码器不能充分解码多模态特征。为了解决上述问题，我们提出了用于视频时刻检索和精彩片段检测任务的LD-DETR模型。具体来说，我们首先将相似度矩阵蒸馏为单位矩阵，以减轻重叠语义信息的影响。然后，我们设计了一种方法，使卷积层能够更高效地提取多模态局部特征。最后，我们将Transformer解码器的输出反馈回其自身，以充分解码多模态信息。我们在四个公开基准上评估了LD-DETR，并进行了大量实验以证明我们方法的优越性和有效性。我们的模型在QVHighlight、Charades-STA和TACoS数据集上优于最先进的模型。我们的代码可在https://github.com/qingchen239/ld-detr获取。
英文摘要：
Video Moment Retrieval and Highlight Detection aim to find corresponding content in the video based on a text query. Existing models usually first use contrastive learning methods to align video and text features, then fuse and extract multimodal information, and finally use a Transformer Decoder to decode multimodal information. However, existing methods face several issues: (1) Overlapping semantic information between different samples in the dataset hinders the model's multimodal aligning performance; (2) Existing models are not able to efficiently extract local features of the video; (3) The Transformer Decoder used by the existing model cannot adequately decode multimodal features. To address the above issues, we proposed the LD-DETR model for Video Moment Retrieval and Highlight Detection tasks. Specifically, we first distilled the similarity matrix into the identity matrix to mitigate the impact of overlapping semantic information. Then, we designed a method that enables convolutional layers to extract multimodal local features more efficiently. Finally, we fed the output of the Transformer Decoder back into itself to adequately decode multimodal information. We evaluated LD-DETR on four public benchmarks and conducted extensive experiments to demonstrate the superiority and effectiveness of our approach. Our model outperforms the State-Of-The-Art models on QVHighlight, Charades-STA and TACoS datasets. Our code is available at https://github.com/qingchen239/ld-detr.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
