FusionAudio-1.2M：面向多模态上下文融合的细粒度音频描述
FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion
AI总结：
本文提出两阶段多模态融合流程，构建含120万条细粒度描述和600万问答对的FusionAudio数据集，并据此训练出对齐与指令跟随能力更强的CLAP音频编码器。
摘要：
高质量、大规模的音频描述对于推进音频理解至关重要，然而当前的自动化方法往往生成的描述缺乏细粒度细节和上下文准确性，主要原因是它们依赖于有限的单模态或浅层多模态信息。受人类听觉感知的启发——人类听觉感知能够熟练地整合跨模态线索并进行复杂的听觉场景分析——我们提出了一种新颖的两阶段自动化流程。该流程首先利用专门的预训练模型提取多样化的上下文线索（例如，语音、音乐、一般声音以及来自关联视频的视觉信息）。随后，一个大语言模型（LLM）综合这些丰富的多模态输入，生成详细且具有上下文感知能力的音频描述。本工作的主要贡献包括：（1）所提出的可扩展的细粒度音频描述生成方法；（2）FusionAudio，一个包含120万条此类详细描述以及600万问答对的新的大规模数据集；（3）利用FusionAudio开发的增强音频模型，具体是一个基于CLAP的音频编码器，具有更优的音频-文本对齐能力和指令跟随能力。本文为更细致、更准确地自动理解复杂音频环境铺平了道路。代码和数据可在 https://github.com/satsuki2486441738/FusionAudio 获取。
英文摘要：
High-quality, large-scale audio captioning is crucial for advancing audio understanding, yet current automated methods often generate captions that lack fine-grained detail and contextual accuracy, primarily due to their reliance on limited unimodal or superficial multimodal information. Drawing inspiration from human auditory perception, which adeptly integrates cross-modal cues and performs sophisticated auditory scene analysis, we introduce a novel two-stage automated pipeline. This pipeline first employs specialized pretrained models to extract diverse contextual cues (e.g., speech, music, general sounds, and visual information from associated video). A large language model (LLM) then synthesizes these rich, multimodal inputs to generate detailed and context-aware audio captions. Key contributions of this work include: (1) the proposed scalable method for fine-grained audio caption generation; (2) FusionAudio, a new large-scale dataset comprising 1.2 million such detailed captions, combined with 6 million QA pairs; and (3) enhanced audio models developed using FusionAudio, specifically a CLAP-based audio encoder with superior audio-text alignment and instruction following. This paper paves the way for more nuanced and accurate automated understanding of complex audio environments. Code and data can be found in https://github.com/satsuki2486441738/FusionAudio.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
