arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.08688cs.CV
收藏

基于相关性在线挖掘的视频检索模型学习

Learning video retrieval models with relevance-aware online mining

Fondazione Bruno Kessler(布鲁诺·凯塞勒基金会);University of Udine(乌迪内大学);Free University of Bozen-Bolzano(博尔扎诺自由大学)

Alex Falcon, Giuseppe Serra, Oswald Lanz

摘要:

由于每小时上传的视频和相关字幕的数量,基于深度学习的跨模态视频检索解决方案正吸引越来越多的关注。一种典型的方法是学习一个联合的文本-视频嵌入空间,其中视频及其相关字幕的相似性被最大化,而所有其他字幕的相似性较低,称为否定。这种方法假设只有数据集中的视频和字幕对是有效的,但是不同的字幕(正字幕)也可能描述其视觉内容,因此其中一些可能会被错误地惩罚。为了解决这个缺点,我们提出了相关性感知否定和肯定挖掘(RANP),它的基础上的语义的否定,提高了他们的选择,同时也增加了其他有效的积极的相似性。我们探讨了这些技术对两个视频-文本数据集:EPIC-Kitchen-100和MSR-VTT的影响。通过使用所提出的技术,我们在nDCG和mAP方面实现了相当大的改进,从而获得了最先进的结果,例如,EPIC-Kitchen-100上的nDCG和mAP分别为+5.3%和3.0%。我们在\url{https://github.com/aranciokov/ranp}上共享代码和预训练模型。

英文摘要:

Due to the amount of videos and related captions uploaded every hour, deep learning-based solutions for cross-modal video retrieval are attracting more and more attention. A typical approach consists in learning a joint text-video embedding space, where the similarity of a video and its associated caption is maximized, whereas a lower similarity is enforced with all the other captions, called negatives. This approach assumes that only the video and caption pairs in the dataset are valid, but different captions - positives - may also describe its visual contents, hence some of them may be wrongly penalized. To address this shortcoming, we propose the Relevance-Aware Negatives and Positives mining (RANP) which, based on the semantics of the negatives, improves their selection while also increasing the similarity of other valid positives. We explore the influence of these techniques on two video-text datasets: EPIC-Kitchens-100 and MSR-VTT. By using the proposed techniques, we achieve considerable improvements in terms of nDCG and mAP, leading to state-of-the-art results, e.g. +5.3% nDCG and +3.0% mAP on EPIC-Kitchens-100. We share code and pretrained models at \url{https://github.com/aranciokov/ranp}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑