用于自动音频字幕生成的离散音频表示
Discrete Audio Representations for Automated Audio Captioning
AI总结：
该研究针对离散音频令牌在自动音频字幕生成中应用不足、无监督令牌化导致性能下降的问题，提出有监督音频令牌器，经Clotho数据集验证其性能优于传统音频令牌。
摘要：
离散音频表示被称为音频令牌（audio token），大致可分为语义令牌和声学令牌，通常通过对连续音频表示进行无监督令牌化生成。然而，它们在自动音频字幕生成（AAC）中的应用仍未得到充分探索。本文通过对多种令牌化方法的对比分析，系统研究了音频令牌驱动模型用于AAC的可行性。我们的研究结果表明，与直接使用连续音频表示的模型相比，音频令牌化会导致AAC模型的性能下降。为解决这一问题，我们提出了一种以音频标注目标训练的有监督音频令牌器。与缺乏显式语义理解的无监督令牌器不同，所提出的令牌器能有效捕捉音频事件信息。在Clotho数据集上开展的实验表明，所提出的音频令牌在AAC任务中优于传统音频令牌。
英文摘要：
Discrete audio representations, termed audio tokens, are broadly categorized into semantic and acoustic tokens, typically generated through unsupervised tokenization of continuous audio representations. However, their applicability to automated audio captioning (AAC) remains underexplored. This paper systematically investigates the viability of audio token-driven models for AAC through comparative analyses of various tokenization methods. Our findings reveal that audio tokenization leads to performance degradation in AAC models compared to those that directly utilize continuous audio representations. To address this issue, we introduce a supervised audio tokenizer trained with an audio tagging objective. Unlike unsupervised tokenizers, which lack explicit semantic understanding, the proposed tokenizer effectively captures audio event information. Experiments conducted on the Clotho dataset demonstrate that the proposed audio tokens outperform conventional audio tokens in the AAC task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
