arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.02665cs.SDcs.LGcs.MMeess.AS
收藏

基于异构图学习的声事件分类

Heterogeneous Graph Learning for Acoustic Event Classification

University of Warwick(华威大学);University of Surrey(萨里大学);Google Research(谷歌研究院);University of Glasgow(格拉斯哥大学)

Amir Shirian, Mona Ahmadian, Krishna Somandepalli, Tanaya Guha

摘要:

异构图提供了一种紧凑、高效和可扩展的方式来对涉及多个不同模态的数据进行建模。这使得使用异构图对视听数据进行建模成为一种有吸引力的选择。然而,图形结构并不自然地出现在视听数据中。视听数据的图表是手动构建的,这既困难又不理想。在这项工作中,我们解决这个问题,(i)提出一个参数化的图构造策略的模态内的边缘,和(ii)学习的跨模态的边缘。为此,我们开发了一个新的模型,异构图跨模态网络(HGCN),学习跨模态边缘。我们提出的模型可以适应各种空间和时间尺度,由于其参数化的建设,而可学习的跨模态边缘有效地连接跨模态的相关节点。在大型基准数据集(AudioSet)上的实验表明,我们的模型是最先进的(平均精度为0.53),优于基于transformer的模型和其他基于图的模型。

英文摘要:

Heterogeneous graphs provide a compact, efficient, and scalable way to model data involving multiple disparate modalities. This makes modeling audiovisual data using heterogeneous graphs an attractive option. However, graph structure does not appear naturally in audiovisual data. Graphs for audiovisual data are constructed manually which is both difficult and sub-optimal. In this work, we address this problem by (i) proposing a parametric graph construction strategy for the intra-modal edges, and (ii) learning the crossmodal edges. To this end, we develop a new model, heterogeneous graph crossmodal network (HGCN) that learns the crossmodal edges. Our proposed model can adapt to various spatial and temporal scales owing to its parametric construction, while the learnable crossmodal edges effectively connect the relevant nodes across modalities. Experiments on a large benchmark dataset (AudioSet) show that our model is state-of-the-art (0.53 mean average precision), outperforming transformer-based models and other graph-based models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑