分解声音和像素,重组事件
Decompose the Sounds and Pixels, Recompose the Events
摘要:
在本文中,我们提出了一个框架,围绕一个新的架构称为事件分解重组网络(EDRNet),以解决视听事件(AVE)的本地化问题,在监督和弱监督设置。真实世界中的AVE表现出共同的解开模式(称为事件进展检查点(EPC)),人类可以通过听觉和视觉的合作来感知。与试图识别整个事件序列的早期方法不同,EDRNet使用堆叠的时间卷积对EPC和EPC间关系进行建模。基于假设EPC表示理论上是一致的事件类别,我们介绍了基于状态机的视频融合,一种新的增强技术,混合源视频使用不同的EPC模板序列。此外,我们设计了一个新的损失函数,称为陆地-海岸-海洋损失,以压缩连续的前景和背景表示。最后,为了减轻在弱监督期间混淆事件的问题,我们提出了一种称为Bag to Instance Label Correction的预测稳定方法。在AVE数据集上的实验表明,我们的集体框架在相当大的幅度上优于最先进的框架。
英文摘要:
In this paper, we propose a framework centering around a novel architecture called the Event Decomposition Recomposition Network (EDRNet) to tackle the Audio-Visual Event (AVE) localization problem in the supervised and weakly supervised settings. AVEs in the real world exhibit common unravelling patterns (termed as Event Progress Checkpoints (EPC)), which humans can perceive through the cooperation of their auditory and visual senses. Unlike earlier methods which attempt to recognize entire event sequences, the EDRNet models EPCs and inter-EPC relationships using stacked temporal convolutions. Based on the postulation that EPC representations are theoretically consistent for an event category, we introduce the State Machine Based Video Fusion, a novel augmentation technique that blends source videos using different EPC template sequences. Additionally, we design a new loss function called the Land-Shore-Sea loss to compactify continuous foreground and background representations. Lastly, to alleviate the issue of confusing events during weak supervision, we propose a prediction stabilization method called Bag to Instance Label Correction. Experiments on the AVE dataset show that our collective framework outperforms the state-of-the-art by a sizable margin.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。