零样本音频描述生成：基于音频-语言模型引导与音频上下文关键词
Zero-shot audio captioning with audio-language model guidance and audio context keywords
AI总结：
提出ZerAuCap框架，利用预训练大语言模型和音频-语言模型引导，结合音频上下文关键词，实现零样本音频描述生成，在AudioCaps和Clotho数据集上取得最优结果。
摘要：
零样本音频描述生成旨在无需针对该任务进行预先训练的情况下，自动为音频内容生成描述性文本。与将包含口语的音频内容转换为文本的语音识别不同，音频描述生成通常关注环境声音或人类执行动作时产生的声音。受零样本图像描述生成方法的启发，我们提出了ZerAuCap，一个新颖的框架，用于将此类通用音频信号总结为文本描述，而无需特定任务的训练。具体而言，我们的框架利用预训练的大语言模型（LLM）生成文本，并由预训练的音频-语言模型引导，以生成描述音频内容的描述。此外，我们使用音频上下文关键词来提示语言模型生成与声音广泛相关的文本。我们提出的框架在AudioCaps和Clotho数据集上的零样本音频描述生成任务中达到了最先进的结果。我们的代码可在https://github.com/ExplainableML/ZerAuCap获取。
英文摘要：
Zero-shot audio captioning aims at automatically generating descriptive textual captions for audio content without prior training for this task. Different from speech recognition which translates audio content that contains spoken language into text, audio captioning is commonly concerned with ambient sounds, or sounds produced by a human performing an action. Inspired by zero-shot image captioning methods, we propose ZerAuCap, a novel framework for summarising such general audio signals in a text caption without requiring task-specific training. In particular, our framework exploits a pre-trained large language model (LLM) for generating the text which is guided by a pre-trained audio-language model to produce captions that describe the audio content. Additionally, we use audio context keywords that prompt the language model to generate text that is broadly relevant to sounds. Our proposed framework achieves state-of-the-art results in zero-shot audio captioning on the AudioCaps and Clotho datasets. Our code is available at https://github.com/ExplainableML/ZerAuCap.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
