arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.05060cs.CV
收藏

AVE-CLIP:用于视听事件定位的AudioCLIP-based多窗口时间Transformer

AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization

The University of Texas at Austin(德克萨斯大学奥斯汀分校)

Tanvir Mahmud, Diana Marculescu

摘要:

视听事件(AVE)由视频片段中视觉和听觉信号的对应关系表示。AVE的精确定位是非常具有挑战性的,因为它需要有效的多模态特征对应以使短距离和长距离的时间相互作用接地。现有的方法在捕捉不同规模的多模态交互由于无效的多模态培训策略的斗争。为了克服这一限制,我们引入了AVE-CLIP,这是一种新的框架,它将在大规模视听数据上预训练的AudioCLIP与多窗口时间Transformer集成在一起,以有效地在不同时间尺度的视频帧上操作。我们的贡献是三方面的:(1)我们引入了一个多阶段的训练框架,通过对比微调,有效的平均视频特征提取和多尺度训练阶段,将使用音频图像对预训练的AudioCLIP纳入视频帧的AVE定位任务。(2)我们提出了一个多域注意力机制,在不同的时间尺度上的时间和特征域上操作,以融合局部和全局特征变化。(3)我们引入了一个时间精炼方案,事件引导的注意力,然后通过一个简单而有效的后处理步骤来处理不同事件背景的显着变化。我们的方法在公开可用的AVE数据集上实现了最先进的性能,平均精度提高了5.9%,这证明了它比现有方法的优越性。

英文摘要:

An audio-visual event (AVE) is denoted by the correspondence of the visual and auditory signals in a video segment. Precise localization of the AVEs is very challenging since it demands effective multi-modal feature correspondence to ground the short and long range temporal interactions. Existing approaches struggle in capturing the different scales of multi-modal interaction due to ineffective multi-modal training strategies. To overcome this limitation, we introduce AVE-CLIP, a novel framework that integrates the AudioCLIP pre-trained on large-scale audio-visual data with a multi-window temporal transformer to effectively operate on different temporal scales of video frames. Our contributions are three-fold: (1) We introduce a multi-stage training framework to incorporate AudioCLIP pre-trained with audio-image pairs into the AVE localization task on video frames through contrastive fine-tuning, effective mean video feature extraction, and multi-scale training phases. (2) We propose a multi-domain attention mechanism that operates on both temporal and feature domains over varying timescales to fuse the local and global feature variations. (3) We introduce a temporal refining scheme with event-guided attention followed by a simple-yet-effective post processing step to handle significant variations of the background over diverse events. Our method achieves state-of-the-art performance on the publicly available AVE dataset with 5.9% mean accuracy improvement which proves its superiority over existing approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑