用于音频字幕生成的音频差异学习
Audio Difference Learning for Audio Captioning
AI总结：
本研究提出音频差异学习训练范式，通过共享编码器提取输入与参考音频的差异特征生成字幕，并采用音频混合技术免除了差异标注需求，在 Clotho 和 ESC50 数据集上使 SPIDEr 分数提升 7%。
摘要：
本研究引入了一种新颖的训练范式——音频差异学习（audio difference learning），以改进音频字幕生成。所提学习方法的基本概念是创建一个保留音频之间关系的特征表示空间，从而能够生成详述复杂音频信息的字幕。该方法采用一段参考音频和输入音频，两者通过共享编码器转换为特征表示。随后从这些差异特征生成字幕，以描述它们之间的差异。此外，还提出了一种独特技术，即将输入音频与附加音频混合，并将附加音频作为参考。这使得混合音频与参考音频之间的差异还原为原始输入音频。因此，原始输入的字幕可作为它们差异的字幕，无需为差异提供额外标注。在使用 Clotho 和 ESC50 数据集的实验中，所提方法与传统方法相比，SPIDEr 分数提高了 7%。
英文摘要：
This study introduces a novel training paradigm, audio difference learning, for improving audio captioning. The fundamental concept of the proposed learning method is to create a feature representation space that preserves the relationship between audio, enabling the generation of captions that detail intricate audio information. This method employs a reference audio along with the input audio, both of which are transformed into feature representations via a shared encoder. Captions are then generated from these differential features to describe their differences. Furthermore, a unique technique is proposed that involves mixing the input audio with additional audio, and using the additional audio as a reference. This results in the difference between the mixed audio and the reference audio reverting back to the original input audio. This allows the original input's caption to be used as the caption for their difference, eliminating the need for additional annotations for the differences. In the experiments using the Clotho and ESC50 datasets, the proposed method demonstrated an improvement in the SPIDEr score by 7% compared to conventional methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
