通过可听性引导实现零样本音频字幕生成
Zero-Shot Audio Captioning via Audibility Guidance
AI总结：
本文提出一种零样本音频字幕生成方法，通过结合GPT-2、ImageBind和文本分类器，在AudioCap数据集上验证了可听性引导能显著提升性能。
摘要：
音频字幕生成任务在本质上与图像和视频字幕生成任务相似。然而，它受到的关注要少得多。我们提出了音频字幕生成的三个期望属性——(i) 生成文本的流畅性，(ii) 生成文本对输入音频的忠实度，以及与之有些相关的 (iii) 可听性，即仅基于音频就能被感知的质量。我们的方法是一种零样本方法，即我们不学习执行字幕生成。相反，字幕生成作为一个推理过程发生，该过程涉及对应于三个期望属性的三个网络：(i) 一个大型语言模型，在我们的案例中，出于便利性原因，使用 GPT-2；(ii) 一个提供音频文件和文本之间匹配分数的模型，为此我们使用了一个名为 ImageBind 的多模态匹配网络；以及 (iii) 一个文本分类器，该分类器使用我们通过指导 GPT-4 自动收集的数据集进行训练，所使用的提示旨在引导生成可听和不可听的句子。我们在 AudioCap 数据集上展示了我们的结果，证明与缺乏此目标的基线相比，可听性引导显著提升了性能。
英文摘要：
The task of audio captioning is similar in essence to tasks such as image and video captioning. However, it has received much less attention. We propose three desiderata for captioning audio -- (i) fluency of the generated text, (ii) faithfulness of the generated text to the input audio, and the somewhat related (iii) audibility, which is the quality of being able to be perceived based only on audio. Our method is a zero-shot method, i.e., we do not learn to perform captioning. Instead, captioning occurs as an inference process that involves three networks that correspond to the three desired qualities: (i) A Large Language Model, in our case, for reasons of convenience, GPT-2, (ii) A model that provides a matching score between an audio file and a text, for which we use a multimodal matching network called ImageBind, and (iii) A text classifier, trained using a dataset we collected automatically by instructing GPT-4 with prompts designed to direct the generation of both audible and inaudible sentences. We present our results on the AudioCap dataset, demonstrating that audibility guidance significantly enhances performance compared to the baseline, which lacks this objective.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
