arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2207.07935cs.SDcs.LGcs.MMeess.AS
收藏

基于异构图的视觉感知声事件检测

Visually-aware Acoustic Event Detection using Heterogeneous Graphs

The University of Warwick(华威大学);Google Research(谷歌研究院);The University of Glasgow(格拉斯哥大学)

Amir Shirian, Krishna Somandepalli, Victor Sanchez, Tanaya Guha

摘要:

听觉事件的感知本质上是多模态的,依赖于听觉和视觉线索。大量现有的多模态方法使用模态特定的模型处理每个模态,然后融合嵌入以编码联合信息。相比之下,我们采用异构图来明确捕获模态之间的空间和时间关系,并表示有关潜在信号的详细信息。使用异构图方法来解决视觉感知的声学事件分类的任务,其作为一种紧凑、高效和可扩展的方式来以图的形式表示数据。通过异构图,我们有效地建模内和模态间的关系在空间和时间尺度。我们的模型可以通过相关的超参数轻松地适应不同尺度的事件。AudioSet,一个大型的基准测试,实验表明,我们的模型达到了最先进的性能。

英文摘要:

Perception of auditory events is inherently multimodal relying on both audio and visual cues. A large number of existing multimodal approaches process each modality using modality-specific models and then fuse the embeddings to encode the joint information. In contrast, we employ heterogeneous graphs to explicitly capture the spatial and temporal relationships between the modalities and represent detailed information about the underlying signal. Using heterogeneous graph approaches to address the task of visually-aware acoustic event classification, which serves as a compact, efficient and scalable way to represent data in the form of graphs. Through heterogeneous graphs, we show efficiently modelling of intra- and inter-modality relationships both at spatial and temporal scales. Our model can easily be adapted to different scales of events through relevant hyperparameters. Experiments on AudioSet, a large benchmark, shows that our model achieves state-of-the-art performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑