arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2112.04771cs.CV
收藏

基于多层稠密差分映射的类事件边界检测

Progressive Attention on Multi-Level Dense Difference Maps for Generic Event Boundary Detection

Nanjing University(南京大学);SenseTime Research(商汤科技研究院)

Jiaqi Tang, Zhaoyang Liu, Chen Qian, Wayne Wu, Limin Wang

摘要:

通用事件边界检测是视频理解中的一个重要而又具有挑战性的任务,其目的是检测人类自然感知事件边界的时刻。这项任务的主要挑战是感知不同事件边界的各种时间变化。为此,本文提出了一个有效的和端到端的可学习的框架(DDM-Net)。为了解决事件边界的多样性和复杂语义,我们做了三个显着的改进。首先,我们构建一个特征库来存储空间和时间的多层次特征,为多尺度下的差异计算做准备。其次,为了减轻以前的方法的时间建模不足,我们提出了密集的差异映射(DDM),以全面表征运动模式。最后,我们利用渐进式注意力的多层次DDM联合聚合外观和运动线索。因此,DDM-Net在Kinetics-GEBD和TAPOS基准测试中分别实现了14%和8%的显着提升,并在没有额外功能的情况下超越了LOVEU Challenge@CVPR 2021的前一名解决方案。最先进的结果表明,更丰富的运动表示和更复杂的聚合,在处理通用事件边界检测的多样性的有效性。代码可在\url{https://github.com/MCG-NJU/DDM}上获得。

英文摘要:

Generic event boundary detection is an important yet challenging task in video understanding, which aims at detecting the moments where humans naturally perceive event boundaries. The main challenge of this task is perceiving various temporal variations of diverse event boundaries. To this end, this paper presents an effective and end-to-end learnable framework (DDM-Net). To tackle the diversity and complicated semantics of event boundaries, we make three notable improvements. First, we construct a feature bank to store multi-level features of space and time, prepared for difference calculation at multiple scales. Second, to alleviate inadequate temporal modeling of previous methods, we present dense difference maps (DDM) to comprehensively characterize the motion pattern. Finally, we exploit progressive attention on multi-level DDM to jointly aggregate appearance and motion clues. As a result, DDM-Net respectively achieves a significant boost of 14% and 8% on Kinetics-GEBD and TAPOS benchmark, and outperforms the top-1 winner solution of LOVEU Challenge@CVPR 2021 without bells and whistles. The state-of-the-art result demonstrates the effectiveness of richer motion representation and more sophisticated aggregation, in handling the diversity of generic event boundary detection. The code is made available at \url{https://github.com/MCG-NJU/DDM}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑