arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.08776cs.CVcs.IR
收藏

一个高效的COarse-to-fiNE比对框架@ Ego4D自然语言比对挑战赛2022

An Efficient COarse-to-fiNE Alignment Framework @ Ego4D Natural Language Queries Challenge 2022

City University of Hong Kong(香港城市大学);Microsoft Research Asia(微软亚洲研究院);National University of Singapore(新加坡国立大学);Beihang University(北京航空航天大学);Singapore Management University(新加坡管理大学);Sun Yat-sen University(中山大学)

Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, Wing-Kwong Chan, Chong-Wah Ngo, Zheng Shou, Nan Duan

摘要:

本技术报告描述了ECCV 2022中Ego 4D自然语言挑战赛(NLQ)的CONE方法。我们利用我们的模型CONE,一个有效的以窗口为中心的粗到细对齐框架。具体来说,CONE通过滑动窗口方法动态地将长视频切片成候选窗口。以窗口为中心,CONE(1)通过对比学习学习窗口间(粗粒度)语义方差,并通过预过滤与NL查询相关的候选窗口来加速推理;(2)利用对比视觉-文本预训练模型EgoVLP强大的多模态对齐能力进行窗口内(细粒度)候选矩排名。在盲测试集上,对于R1@IoU=0.3和R1@IoU=0.5,CONE分别达到15.26和9.24。

英文摘要:

This technical report describes the CONE approach for Ego4D Natural Language Queries (NLQ) Challenge in ECCV 2022. We leverage our model CONE, an efficient window-centric COarse-to-fiNE alignment framework. Specifically, CONE dynamically slices the long video into candidate windows via a sliding window approach. Centering at windows, CONE (1) learns the inter-window (coarse-grained) semantic variance through contrastive learning and speeds up inference by pre-filtering the candidate windows relevant to the NL query, and (2) conducts intra-window (fine-grained) candidate moments ranking utilizing the powerful multi-modal alignment ability of the contrastive vision-text pre-trained model EgoVLP. On the blind test set, CONE achieves 15.26 and 9.24 for R1@IoU=0.3 and R1@IoU=0.5, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑