arXiv 2305.187532023-05-30eess.AScs.SD

基于双 Transformer 解码器特征融合网络的自动音频描述

Dual Transformer Decoder based Features Fusion Network for Automated Audio Captioning

University of Surrey（萨里大学）；Izmir Katip Celebi University（伊兹密尔卡蒂普·切莱比大学）

Jianyuan Sun, Xubo Liu, Xinhao Mei, Volkan Kılıç, Mark D. Plumbley, Wenwu Wang

备注：: INTERSPEECH 2023. arXiv admin note: substantial text overlap with arXiv:2210.05037

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 LHDFF 模型，通过残差 PANNs 编码器融合低维与高维音频特征，并采用双 Transformer 解码器并行生成描述，实验表明其性能优于现有音频描述模型。

摘要：

自动音频描述（AAC）旨在生成音频内容的文本描述。现有 AAC 模型虽然取得了良好效果，但仅使用编码器的高维表示。由于高维表示包含大量信息，高维方法往往存在信息学习不充分的问题。本文提出了一种新的编码器-解码器模型，称为低维与高维特征融合（LHDFF）。LHDFF 使用一种新的 PANNs 编码器，称为残差 PANNs（RPANNs），来融合低维和高维特征。低维特征包含关于特定音频场景的有限信息。低维与高维特征的融合可以通过反复强调特定音频场景信息来提升模型性能。为了充分利用融合后的特征，LHDFF 采用双 Transformer 解码器结构并行生成描述。实验结果表明，LHDFF 优于现有的音频描述模型。

英文摘要：

Automated audio captioning (AAC) which generates textual descriptions of audio content. Existing AAC models achieve good results but only use the high-dimensional representation of the encoder. There is always insufficient information learning of high-dimensional methods owing to high-dimensional representations having a large amount of information. In this paper, a new encoder-decoder model called the Low- and High-Dimensional Feature Fusion (LHDFF) is proposed. LHDFF uses a new PANNs encoder called Residual PANNs (RPANNs) to fuse low- and high-dimensional features. Low-dimensional features contain limited information about specific audio scenes. The fusion of low- and high-dimensional features can improve model performance by repeatedly emphasizing specific audio scene information. To fully exploit the fused features, LHDFF uses a dual transformer decoder structure to generate captions in parallel. Experimental results show that LHDFF outperforms existing audio captioning models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。