无需音频训练音频字幕模型
Training Audio Captioning Models without Audio
AI总结：
本研究提出仅使用文本训练自动音频字幕生成系统的方法，利用CLAP多模态空间并结合噪声注入或可学习适配器弥合模态鸿沟，性能媲美配对音频训练的先进模型。
摘要：
自动音频字幕生成（Automated Audio Captioning, AAC）是指为给定的音频流生成自然语言描述的任务。典型的AAC系统需要人工整理的音频片段及相应文本字幕标注的训练数据。创建这些音频-字幕对成本高昂，导致该任务普遍存在数据稀缺问题。在本研究中，我们解决了这一主要局限性，并提出了一种仅使用文本训练AAC系统的方法。该方法利用了对比训练的音频-文本模型（如CLAP）的多模态空间。在训练期间，解码器根据预训练的CLAP文本编码器生成字幕。在推理期间，文本编码器被预训练的CLAP音频编码器替换。为了弥合文本与音频嵌入之间的模态鸿沟，我们提出在训练期间使用噪声注入或可学习适配器。我们发现，所提出的纯文本框架与使用配对音频训练的最先进模型表现相当，表明高效的文本到音频迁移是可行的。最后，我们在无需音频或人工创建文本字幕进行训练的情况下，展示了风格化音频字幕生成和字幕增强。
英文摘要：
Automated Audio Captioning (AAC) is the task of generating natural language descriptions given an audio stream. A typical AAC system requires manually curated training data of audio segments and corresponding text caption annotations. The creation of these audio-caption pairs is costly, resulting in general data scarcity for the task. In this work, we address this major limitation and propose an approach to train AAC systems using only text. Our approach leverages the multimodal space of contrastively trained audio-text models, such as CLAP. During training, a decoder generates captions conditioned on the pretrained CLAP text encoder. During inference, the text encoder is replaced with the pretrained CLAP audio encoder. To bridge the modality gap between text and audio embeddings, we propose the use of noise injection or a learnable adapter, during training. We find that the proposed text-only framework performs competitively with state-of-the-art models trained with paired audio, showing that efficient text-to-audio transfer is possible. Finally, we showcase both stylized audio captioning and caption enrichment while training without audio or human-created text captions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
