arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.02728cs.CV
收藏

面向视频时刻检索的背景感知时刻检测

Background-aware Moment Detection for Video Moment Retrieval

Seoul National University(首尔大学);NAVER AI Lab(NAVER AI实验室)

Minjoon Jung, Youwon Jang, Seongho Choi, Joochan Kim, Jin-Hwa Kim, Byoung-Tak Zhang

AI总结:

针对视频时刻检索中的弱对齐问题,本文提出背景感知时刻检测Transformer(BM-DETR),通过对比学习利用正负查询的联合概率预测目标时刻,有效利用背景信息提升对齐性能,并在四个基准上验证了有效性。

摘要:

视频时刻检索(VMR)旨在根据给定的自然语言查询,在未剪辑视频中定位特定时刻。该任务容易受到视频数据集中固有的弱对齐问题的影响。由于歧义性,查询可能无法完全覆盖相应时刻的相关细节,或者该时刻可能包含未对齐和不相关的帧,从而可能限制性能的进一步提升。为了解决这一问题,我们提出了一种背景感知时刻检测Transformer(BM-DETR)。我们的模型采用对比学习方法,精心利用与视频中其他时刻匹配的负查询。具体来说,我们的模型学习在给定正查询和负查询补集的条件下,根据每一帧的联合概率来预测目标时刻。这使得模型能够有效利用周围背景,提高时刻敏感性并增强视频中的整体对齐。在四个基准上的大量实验证明了我们方法的有效性。我们的代码可在:\url{https://github.com/minjoong507/BM-DETR}获取。

英文摘要:

Video moment retrieval (VMR) identifies a specific moment in an untrimmed video for a given natural language query. This task is prone to suffer the weak alignment problem innate in video datasets. Due to the ambiguity, a query does not fully cover the relevant details of the corresponding moment, or the moment may contain misaligned and irrelevant frames, potentially limiting further performance gains. To tackle this problem, we propose a background-aware moment detection transformer (BM-DETR). Our model adopts a contrastive approach, carefully utilizing the negative queries matched to other moments in the video. Specifically, our model learns to predict the target moment from the joint probability of each frame given the positive query and the complement of negative queries. This leads to effective use of the surrounding background, improving moment sensitivity and enhancing overall alignments in videos. Extensive experiments on four benchmarks demonstrate the effectiveness of our approach. Our code is available at: \url{https://github.com/minjoong507/BM-DETR}

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑