arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.02504cs.CVcs.AI
收藏

观看视频,捕捉关键词:用于时刻检索和高光检测的上下文感知关键词注意力

Watch Video, Catch Keyword: Context-aware Keyword Attention for Moment Retrieval and Highlight Detection

Kyung Hee University(庆熙大学);Sungkyunkwan University(成均馆大学)

Sung Jin Um, Dongjin Kim, Sangmin Lee, Jung Uk Kim

AI总结:

针对视频时刻检索与高光检测难以利用全局上下文判断关键词相关性的问题,提出上下文感知关键词注意力及聚类、对比学习模块,在三个基准上显著提升性能。

摘要:

视频时刻检索和高光检测的目标是根据给定文本查询识别特定片段和高光。随着视频内容的快速增长以及这些任务之间的重叠,近期研究同时处理两者。然而,它们仍难以充分捕捉整体视频上下文,导致判断哪些词最相关具有挑战性。本文提出一种新颖的 Video Context-aware Keyword Attention(视频上下文感知关键词注意力)模块,通过在整个视频上下文中捕捉关键词变化来克服这一限制。为此,我们引入视频上下文聚类模块,提供整体视频上下文的简洁表示,从而增强对关键词动态的理解。此外,我们提出带有 keyword-aware contrastive learning(关键词感知对比学习)的关键词权重检测模块,该模块融合关键词信息以增强视觉与文本特征之间的细粒度对齐。在 QVHighlights、TVSum 和 Charades-STA 基准上的大量实验表明,与现有方法相比,所提方法显著提升了时刻检索和高光检测任务的性能。代码位于:https://github.com/VisualAIKHU/Keyword-DETR

英文摘要:

The goal of video moment retrieval and highlight detection is to identify specific segments and highlights based on a given text query. With the rapid growth of video content and the overlap between these tasks, recent works have addressed both simultaneously. However, they still struggle to fully capture the overall video context, making it challenging to determine which words are most relevant. In this paper, we present a novel Video Context-aware Keyword Attention module that overcomes this limitation by capturing keyword variation within the context of the entire video. To achieve this, we introduce a video context clustering module that provides concise representations of the overall video context, thereby enhancing the understanding of keyword dynamics. Furthermore, we propose a keyword weight detection module with keyword-aware contrastive learning that incorporates keyword information to enhance fine-grained alignment between visual and textual features. Extensive experiments on the QVHighlights, TVSum, and Charades-STA benchmarks demonstrate that our proposed method significantly improves performance in moment retrieval and highlight detection tasks compared to existing approaches. Our code is available at: https://github.com/VisualAIKHU/Keyword-DETR

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑