arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.02178cs.CVcs.CL
收藏

基于现成模型的Zero-shot视频矩检索

Zero-shot Video Moment Retrieval With Off-the-Shelf Models

The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Anuj Diwan, Puyuan Peng, Raymond J. Mooney

摘要:

对于大多数机器学习社区来说,收集高质量人工注释数据的昂贵性以及无法在有限的计算上有效地微调非常大的最先进的预训练模型是为新任务构建模型的主要瓶颈。我们提出了一个zero-shot简单的方法,这样的任务,视频时刻检索(VMR),不执行任何额外的微调,只是重新使用现成的模型训练其他任务。我们的三步方法包括矩建议,矩查询匹配和后处理,所有这些都只使用现成的模型。在VMR的QVHighlights基准测试中,我们在所有指标上将以前的zero-shot方法的性能提高了至少2.5倍,并将zero-shot和最先进的监督之间的差距缩小了74%以上。此外,我们还表明,我们的zero-shot方法在召回指标上击败了非预训练的监督模型,并且在mAP指标上非常接近;并且在较短的时刻上,它的表现也优于最好的预训练监督模型。最后,我们消融和分析我们的结果,并提出有趣的未来方向。

英文摘要:

For the majority of the machine learning community, the expensive nature of collecting high-quality human-annotated data and the inability to efficiently finetune very large state-of-the-art pretrained models on limited compute are major bottlenecks for building models for new tasks. We propose a zero-shot simple approach for one such task, Video Moment Retrieval (VMR), that does not perform any additional finetuning and simply repurposes off-the-shelf models trained on other tasks. Our three-step approach consists of moment proposal, moment-query matching and postprocessing, all using only off-the-shelf models. On the QVHighlights benchmark for VMR, we vastly improve performance of previous zero-shot approaches by at least 2.5x on all metrics and reduce the gap between zero-shot and state-of-the-art supervised by over 74%. Further, we also show that our zero-shot approach beats non-pretrained supervised models on the Recall metrics and comes very close on mAP metrics; and that it also performs better than the best pretrained supervised model on shorter moments. Finally, we ablate and analyze our results and propose interesting future directions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑