arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.10918cs.CVcs.CLcs.IR
收藏

CONE:一种用于长视频时间定位的高效由粗到细对齐框架

CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding

City University of Hong Kong(香港城市大学);Microsoft Research Asia(微软亚洲研究院);National University of Singapore(新加坡国立大学);Beihang University(北京航空航天大学);Singapore Management University(新加坡管理大学);Sun Yat-sen University(中山大学)

Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, Wing-Kwong Chan, Chong-Wah Ngo, Zheng Shou, Nan Duan

AI总结:

本文提出CONE框架,通过查询引导窗口选择和由粗到细对比学习机制,在保持SOTA结果的同时显著提升长视频时间定位的推理效率与多模态对齐性能。

摘要:

本文解决了一个新兴且具有挑战性的长视频时间定位(VTG)问题,该问题旨在定位与自然语言(NL)查询相关的视频片段。与短视频相比,长视频同样需求旺盛但探索较少,这带来了更高的推理计算成本和更弱的多模态对齐等新挑战。为应对这些挑战,我们提出了CONE,一个高效的由粗到细(COarse-to-fiNE)对齐框架。CONE是一个即插即用框架,可叠加在现有VTG模型之上,通过滑动窗口机制处理长视频。具体而言,CONE(1)引入了一种查询引导的窗口选择策略以加速推理,(2)通过新颖地引入对比学习提出了一种由粗到细机制,以增强长视频的多模态对齐。在两个大规模长视频VTG基准上的大量实验一致表明,该方法既取得了显著的性能提升(例如在MAD上从3.13%提升到6.87%),也达到了最先进的结果。分析还揭示了更高的效率,因为查询引导的窗口选择机制在Ego4D-NLQ上将推理时间加速了2倍,在MAD上加速了15倍,同时保持了SOTA结果。代码已在https://github.com/houzhijian/CONE发布。

英文摘要:

This paper tackles an emerging and challenging problem of long video temporal grounding~(VTG) that localizes video moments related to a natural language (NL) query. Compared with short videos, long videos are also highly demanded but less explored, which brings new challenges in higher inference computation cost and weaker multi-modal alignment. To address these challenges, we propose CONE, an efficient COarse-to-fiNE alignment framework. CONE is a plug-and-play framework on top of existing VTG models to handle long videos through a sliding window mechanism. Specifically, CONE (1) introduces a query-guided window selection strategy to speed up inference, and (2) proposes a coarse-to-fine mechanism via a novel incorporation of contrastive learning to enhance multi-modal alignment for long videos. Extensive experiments on two large-scale long VTG benchmarks consistently show both substantial performance gains (e.g., from 3.13% to 6.87% on MAD) and state-of-the-art results. Analyses also reveal higher efficiency as the query-guided window selection mechanism accelerates inference time by 2x on Ego4D-NLQ and 15x on MAD while keeping SOTA results. Codes have been released at https://github.com/houzhijian/CONE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑