arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.14441cs.CVcs.AI
收藏

Attend-Fusion:用于视频分类的高效音视频融合方法

Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification

University of Surrey(萨里大学);Centre for Vision, Speech and Signal Processing (CVSSP)(视觉、语音与信号处理中心)

Mahrukh Awan, Asmar Nadeem, Muhammad Junaid Awan, Armin Mustafa, Syed Sameed Husain

AI总结:

提出紧凑的音视频融合模型Attend-Fusion,在YouTube-8M数据集上仅用72M参数即达到75.64%的F1分数,性能媲美341M参数的大型基线模型,为资源受限环境下的视频理解提供了高效方案。

摘要:

利用音频和视觉两种模态进行视频分类是一项具有挑战性的任务,因为现有方法需要大型模型架构,导致高计算复杂度和资源需求。而较小的架构则难以达到最佳性能。在本文中,我们提出了Attend-Fusion,这是一种音视频(AV)融合方法,引入了专门设计用于捕捉视频数据中复杂音视频关系的紧凑模型架构。通过在具有挑战性的YouTube-8M数据集上进行大量实验,我们证明Attend-Fusion仅用72M参数就达到了75.64%的F1分数,这与全连接后期融合(Fully-Connected Late Fusion)等较大基线模型的性能(75.96%的F1分数,341M参数)相当。Attend-Fusion在实现与较大基线模型相似性能的同时,将模型大小减少了近80%,突显了其在模型复杂度方面的高效性。我们的工作表明,Attend-Fusion模型有效地结合了音频和视觉信息进行视频分类,在大幅减小模型规模的同时实现了具有竞争力的性能。该方法为在各种应用的资源受限环境中部署高性能视频理解系统开辟了新的可能性。

英文摘要:

Exploiting both audio and visual modalities for video classification is a challenging task, as the existing methods require large model architectures, leading to high computational complexity and resource requirements. Smaller architectures, on the other hand, struggle to achieve optimal performance. In this paper, we propose Attend-Fusion, an audio-visual (AV) fusion approach that introduces a compact model architecture specifically designed to capture intricate audio-visual relationships in video data. Through extensive experiments on the challenging YouTube-8M dataset, we demonstrate that Attend-Fusion achieves an F1 score of 75.64\% with only 72M parameters, which is comparable to the performance of larger baseline models such as Fully-Connected Late Fusion (75.96\% F1 score, 341M parameters). Attend-Fusion achieves similar performance to the larger baseline model while reducing the model size by nearly 80\%, highlighting its efficiency in terms of model complexity. Our work demonstrates that the Attend-Fusion model effectively combines audio and visual information for video classification, achieving competitive performance with significantly reduced model size. This approach opens new possibilities for deploying high-performance video understanding systems in resource-constrained environments across various applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑