用于自动音频字幕生成的Prefix tuning
Prefix tuning for automated audio captioning
AI总结：
针对音频字幕生成中训练数据缺乏导致的泛化难题，提出利用冻结的预训练语言模型，通过映射网络将音频特征转换为prefixes，在Clotho和AudioCaps数据集上优于现有方法。
摘要：
音频字幕生成旨在根据环境声音生成文本描述。由于缺乏音频-文本配对训练数据，泛化困难是音频字幕生成的一项挑战。在本研究中，我们提出了一种简单有效的方法，通过利用预训练语言模型来处理小规模数据集。我们保持语言模型冻结以维持文本生成的表达能力，仅学习从输入音频中提取全局和时间特征。为弥合音频特征与语言模型之间的模态鸿沟，我们采用映射网络将音频特征转换为语言模型能够理解的连续向量，称为prefixes。我们在Clotho和AudioCaps数据集上评估了所提出的方法，结果表明该方法在多种实验设置中均优于现有技术。
英文摘要：
Audio captioning aims to generate text descriptions from environmental sounds. One challenge of audio captioning is the difficulty of the generalization due to the lack of audio-text paired training data. In this work, we propose a simple yet effective method of dealing with small-scaled datasets by leveraging a pre-trained language model. We keep the language model frozen to maintain the expressivity for text generation, and we only learn to extract global and temporal features from the input audio. To bridge a modality gap between the audio features and the language model, we employ mapping networks that translate audio features to the continuous vectors the language model can understand, called prefixes. We evaluate our proposed method on the Clotho and AudioCaps dataset and show our method outperforms prior arts in diverse experimental settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
