用于自动音频字幕生成的图注意力机制
Graph Attention for Automated Audio Captioning
AI总结：
本文提出一种用于自动音频字幕生成的图注意力方法，通过在PANNs编码器后引入图注意力模块学习音频特征的长时上下文关联，有效提升了字幕生成性能。
摘要：
最先进的音频字幕生成方法通常采用编码器-解码器结构，并使用预训练音频神经网络（PANNs）作为编码器进行特征提取。然而，PANNs中使用的卷积操作在捕捉音频信号内的长时依赖性方面存在局限，从而导致音频字幕生成性能的潜在下降。本文提出了一种使用图注意力（GraphAC）的基于编码器-解码器的音频字幕生成新方法。在编码器中，在PANNs之后引入图注意力模块，通过邻接图学习上下文关联（即不同时间帧上音频特征之间的依赖关系），并使用top-k掩码来减轻噪声节点的干扰。学习到的上下文关联通过特征节点聚合产生更有效的特征表示。因此，解码器能够基于从音频信号中学习到的上下文关联，预测关于声学场景和事件的重要语义信息。实验结果表明，得益于将图注意力模块纳入编码器以捕捉音频信号内的长时依赖性，GraphAC优于以PANNs作为编码器的最先进方法。源代码可在https://github.com/LittleFlyingSheep/GraphAC获取。
英文摘要：
State-of-the-art audio captioning methods typically use the encoder-decoder structure with pretrained audio neural networks (PANNs) as encoders for feature extraction. However, the convolution operation used in PANNs is limited in capturing the long-time dependencies within an audio signal, thereby leading to potential performance degradation in audio captioning. This letter presents a novel method using graph attention (GraphAC) for encoder-decoder based audio captioning. In the encoder, a graph attention module is introduced after the PANNs to learn contextual association (i.e. the dependency among the audio features over different time frames) through an adjacency graph, and a top-k mask is used to mitigate the interference from noisy nodes. The learnt contextual association leads to a more effective feature representation with feature node aggregation. As a result, the decoder can predict important semantic information about the acoustic scene and events based on the contextual associations learned from the audio signal. Experimental results show that GraphAC outperforms the state-of-the-art methods with PANNs as the encoders, thanks to the incorporation of the graph attention module into the encoder for capturing the long-time dependencies within the audio signal. The source code is available at https://github.com/LittleFlyingSheep/GraphAC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
