BLAT：基于AudioSet标签引导合成数据的自举式语言-音频预训练
BLAT: Bootstrapping Language-Audio Pre-training based on AudioSet Tag-guided Synthetic Data
AI总结：
针对音频-文本预训练平行数据不足的问题，提出BLAT方法，通过AudioSet标签引导的音频字幕生成构建高质量大规模平行数据，在多数下游任务上取得SOTA零样本分类性能。
摘要：
与充足的视觉-文本预训练研究相比，音频-文本预训练的相关探索较少，主要原因是缺乏足够的平行音频-文本数据。现有多数方法将视觉模态作为音频-文本预训练的枢纽，这不可避免地会引入数据噪声。本文提出利用音频字幕生成技术直接从音频生成文本，无需借助视觉模态，从而消除模态失配带来的潜在噪声。此外，我们提出在AudioSet标签的引导下生成字幕，以得到更准确的字幕内容。通过上述两项改进，我们构建了高质量、大规模的平行音频-文本数据集，并基于该数据集开展音频-文本预训练。我们在一系列下游音频相关任务上全面验证了预训练模型的性能，包括音频分类、音频标注等单模态任务，以及音频-文本检索、基于音频的文本生成等跨模态任务。实验结果表明，该方法在多数数据集上取得了当前最优的零样本分类性能，证明了合成数据的有效性。经过微调后，音频编码器还可作为高效的模式识别模型应用于音频相关任务。合成数据与预训练模型已在线公开。
英文摘要：
Compared with ample visual-text pre-training research, few works explore audio-text pre-training, mostly due to the lack of sufficient parallel audio-text data. Most existing methods incorporate the visual modality as a pivot for audio-text pre-training, which inevitably induces data noise. In this paper, we propose to utilize audio captioning to generate text directly from audio, without the aid of the visual modality so that potential noise from modality mismatch is eliminated. Furthermore, we propose caption generation under the guidance of AudioSet tags, leading to more accurate captions. With the above two improvements, we curate high-quality, large-scale parallel audio-text data, based on which we perform audio-text pre-training. We comprehensively demonstrate the performance of the pre-trained model on a series of downstream audio-related tasks, including single-modality tasks like audio classification and tagging, as well as cross-modal tasks consisting of audio-text retrieval and audio-based text generation. Experimental results indicate that our approach achieves state-of-the-art zero-shot classification performance on most datasets, suggesting the effectiveness of our synthetic data. The audio encoder also serves as an efficient pattern recognition model by fine-tuning it on audio-related tasks. Synthetic data and pre-trained models are available online.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
