arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.09401cs.CL
收藏

基于扩散模型实现多样且高效的音频字幕生成

Towards Diverse and Efficient Audio Captioning via Diffusion Models

Tencent AI Lab(腾讯人工智能实验室);Beijing Institute of Technology(北京理工大学);Chinese Academy of Sciences(中国科学院);University of California, Berkeley(加州大学伯克利分校)

Manjie Xu, Chenxing Li, Xinyi Tu, Yong Ren, Ruibo Fu, Wei Liang, Dong Yu

AI总结:

提出基于扩散的非自回归音频字幕生成模型DAC,解决传统模型生成速度慢、多样性不足的问题,在字幕质量、速度和多样性上均达SOTA,推动跨模态统一生成。

摘要:

我们引入了基于扩散的音频字幕生成(DAC),这是一种专为多样且高效的音频字幕生成定制的非自回归扩散模型。尽管依赖语言骨干网络的现有字幕生成模型在各种字幕生成任务中取得了显著成功,但其在生成速度和多样性方面的不足阻碍了音频理解和多媒体应用的发展。我们基于扩散的框架凭借其固有的随机性和整体上下文建模在字幕生成中提供了独特优势。通过严格评估,我们证明DAC不仅在字幕质量上与现有基准相比达到了SOTA性能水平,而且在生成速度和多样性方面也显著优于它们。DAC的成功表明,文本生成也可以使用扩散骨干网络与音频和视觉生成任务无缝集成,为跨不同模态的统一、音频相关生成模型铺平了道路。

英文摘要:

We introduce Diffusion-based Audio Captioning (DAC), a non-autoregressive diffusion model tailored for diverse and efficient audio captioning. Although existing captioning models relying on language backbones have achieved remarkable success in various captioning tasks, their insufficient performance in terms of generation speed and diversity impede progress in audio understanding and multimedia applications. Our diffusion-based framework offers unique advantages stemming from its inherent stochasticity and holistic context modeling in captioning. Through rigorous evaluation, we demonstrate that DAC not only achieves SOTA performance levels compared to existing benchmarks in the caption quality, but also significantly outperforms them in terms of generation speed and diversity. The success of DAC illustrates that text generation can also be seamlessly integrated with audio and visual generation tasks using a diffusion backbone, paving the way for a unified, audio-related generative model across different modalities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑