通过具有优化音频编码的大型语言模型增强自动音频字幕生成
Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding
AI总结：
本文通过使用CED音频编码器与Q-Former压缩声学token、7B参数的Llama 2解码器及辅助LLM纠错，并结合LoRA优化，有效提升了自动音频字幕生成性能，SPIDEr-FL分数达33.0。
摘要：
自动音频字幕生成（AAC）是一项以自然语言描述音频内容的音频到文本任务。近期，大型语言模型（LLMs）的进步以及音频编码器训练方法的改进，为提升AAC开辟了新可能。因此，我们从三个方面探索增强AAC：1）使用通过一致性集成蒸馏（CED）预训练的音频编码器来提升声学token的有效性，并利用查询变换器（Q-Former）弥合与LLM的模态鸿沟并压缩声学token；2）研究使用具有7B参数的Llama 2作为解码器的优势；3）利用另一个预训练LLM纠正由训练数据不足和标注歧义引起的文本错误。音频编码器和文本解码器均通过低秩适应（LoRA）进行优化。实验表明，每项增强措施均有效。我们的方法获得了33.0的SPIDEr-FL分数，超越了DCASE 2023 Task 6A的获胜者。
英文摘要：
Automated audio captioning (AAC) is an audio-to-text task to describe audio contents in natural language. Recently, the advancements in large language models (LLMs), with improvements in training approaches for audio encoders, have opened up possibilities for improving AAC. Thus, we explore enhancing AAC from three aspects: 1) a pre-trained audio encoder via consistent ensemble distillation (CED) is used to improve the effectivity of acoustic tokens, with a querying transformer (Q-Former) bridging the modality gap to LLM and compress acoustic tokens; 2) we investigate the advantages of using a Llama 2 with 7B parameters as the decoder; 3) another pre-trained LLM corrects text errors caused by insufficient training data and annotation ambiguities. Both the audio encoder and text decoder are optimized by low-rank adaptation (LoRA). Experiments show that each of these enhancements is effective. Our method obtains a 33.0 SPIDEr-FL score, outperforming the winner of DCASE 2023 Task 6A.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
