RECAP：检索增强的音频字幕生成
RECAP: Retrieval-Augmented Audio Captioning
AI总结：
RECAP 提出一种检索增强的音频字幕生成方法，利用 CLAP 检索相似字幕构建提示并借助 GPT-2 与交叉注意力生成字幕，无需微调即可跨域迁移，并在域外场景和新型音频事件上取得显著提升。
摘要：
我们提出 RECAP（REtrieval-Augmented Audio CAPtioning，检索增强的音频字幕生成），这是一种新颖且有效的音频字幕生成系统，该系统以输入音频以及从数据存储中检索到的与该音频相似的其他字幕为条件来生成字幕。此外，我们提出的方法无需任何额外微调即可迁移到任意领域。为了给一个音频样本生成字幕，我们利用音频-文本模型 CLAP 从可替换的数据存储中检索与其相似的字幕，然后用这些字幕构建提示。接下来，我们将该提示输入 GPT-2 解码器，并在 CLAP 编码器和 GPT-2 之间引入交叉注意力层，以音频为条件进行字幕生成。在两个基准数据集 Clotho 和 AudioCaps 上的实验表明，RECAP 在域内设置下取得了具有竞争力的性能，并在域外设置下取得了显著改进。此外，由于 RECAP 能够以无需训练的方式利用仅包含大量文本字幕的数据存储，它展现出独特的能力，能够为训练期间从未见过的新颖音频事件以及包含多个事件的组合音频生成字幕。为了促进该领域的研究，我们还为 AudioSet、AudioCaps 和 Clotho 发布了超过 150,000 条新的弱标注字幕。
英文摘要：
We present RECAP (REtrieval-Augmented Audio CAPtioning), a novel and effective audio captioning system that generates captions conditioned on an input audio and other captions similar to the audio retrieved from a datastore. Additionally, our proposed method can transfer to any domain without the need for any additional fine-tuning. To generate a caption for an audio sample, we leverage an audio-text model CLAP to retrieve captions similar to it from a replaceable datastore, which are then used to construct a prompt. Next, we feed this prompt to a GPT-2 decoder and introduce cross-attention layers between the CLAP encoder and GPT-2 to condition the audio for caption generation. Experiments on two benchmark datasets, Clotho and AudioCaps, show that RECAP achieves competitive performance in in-domain settings and significant improvements in out-of-domain settings. Additionally, due to its capability to exploit a large text-captions-only datastore in a training-free fashion, RECAP shows unique capabilities of captioning novel audio events never seen during training and compositional audios with multiple events. To promote research in this space, we also release 150,000+ new weakly labeled captions for AudioSet, AudioCaps, and Clotho.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
