WavCaps：一个由ChatGPT辅助的弱标注音频字幕数据集，用于音频-语言多模态研究
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
AI总结：
为解决音频-语言多模态数据稀缺问题，我们提出WavCaps，一个含约40万弱标注音频片段的大规模数据集，利用ChatGPT辅助三阶段流程过滤噪声并生成高质量字幕，显著提升下游任务性能。
摘要：
近年来，音频-语言（AL）多模态学习任务的进展显著。然而，由于现有音频-语言数据集的收集过程成本高昂且耗时，且规模有限，研究人员面临挑战。为解决这一数据稀缺问题，我们引入了WavCaps，这是首个大规模弱标注音频字幕数据集，包含约40万个带有配对字幕的音频片段。我们从网络来源和一个声音事件检测数据集中获取音频片段及其原始描述。然而，在线收集的原始描述噪声很大，不适合直接用于自动音频字幕等任务。为克服这一问题，我们提出一个三阶段处理流程，用于过滤噪声数据并生成高质量字幕，其中利用大型语言模型ChatGPT自动过滤和转换原始描述。我们对WavCaps数据集的特征进行了全面分析，并在多个下游音频-语言多模态学习任务上对其进行了评估。基于WavCaps训练的系统显著优于之前的先进（SOTA）模型。我们希望我们提出的WavCaps数据集能促进音频-语言多模态学习的研究，并展示利用ChatGPT增强学术研究的潜力。我们的数据集和代码可在https://github.com/XinhaoMei/WavCaps获取。
英文摘要：
The advancement of audio-language (AL) multimodal learning tasks has been significant in recent years. However, researchers face challenges due to the costly and time-consuming collection process of existing audio-language datasets, which are limited in size. To address this data scarcity issue, we introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions. We sourced audio clips and their raw descriptions from web sources and a sound event detection dataset. However, the online-harvested raw descriptions are highly noisy and unsuitable for direct use in tasks such as automated audio captioning. To overcome this issue, we propose a three-stage processing pipeline for filtering noisy data and generating high-quality captions, where ChatGPT, a large language model, is leveraged to filter and transform raw descriptions automatically. We conduct a comprehensive analysis of the characteristics of WavCaps dataset and evaluate it on multiple downstream audio-language multimodal learning tasks. The systems trained on WavCaps outperform previous state-of-the-art (SOTA) models by a significant margin. Our aspiration is for the WavCaps dataset we have proposed to facilitate research in audio-language multimodal learning and demonstrate the potential of utilizing ChatGPT to enhance academic research. Our dataset and codes are available at https://github.com/XinhaoMei/WavCaps.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
