Bagpiper: 通过丰富描述解决开放式音频任务
Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
AI总结:
提出Bagpiper,一个8B参数音频基础模型,通过丰富描述(自然语言描述)实现音频理解和生成的统一,在MMAU和AIRBench上超越Qwen-2.5-Omni,生成质量优于CosyVoice3和TangoFlux。
摘要:
当前的音频基础模型通常依赖于僵化的、任务特定的监督,处理音频的孤立因素而非整体。相比之下,人类智能整体地处理音频,无缝地将物理信号与抽象认知概念联系起来以执行复杂任务。基于这一理念,我们引入了Bagpiper,一个8B音频基础模型,通过丰富描述(即全面的自然语言描述,封装信号中关键的认知概念,如转录、音频事件)来解释物理音频。通过在600B token的大规模语料库上进行预训练,该模型建立了原始音频与这一高层概念空间之间的稳健双向映射。在微调过程中,Bagpiper采用先描述后处理的流程,模拟中间认知推理步骤,以解决各种任务而无需任务特定的先验知识。实验上,Bagpiper在MMAU和AIRBench音频理解任务上优于Qwen-2.5-Omni,在生成质量上超越CosyVoice3和TangoFlux,能够合成语音、音乐和音效的任意组合。据我们所知,Bagpiper是实现通用音频统一理解与生成的先驱工作之一。模型、数据和代码可在Bagpiper主页获取。
英文摘要:
Current audio foundation models typically rely on rigid, task-specific supervision (e.g., speech recognition), addressing isolated factors of audio rather than the whole. In contrast, human processes audio holistically, seamlessly bridging raw audio waveform with abstract cognitive concepts (e.g., all perception details of audio events) to execute complex tasks. Grounded in this philosophy, we introduce Bagpiper, an 8B audio foundation model that interprets physical audio via rich captions, i.e., comprehensive natural language descriptions that encapsulate the critical cognitive concepts inherent in the audio. By pre-training on a massive corpus of 600B tokens, the model establishes a robust bidirectional mapping between raw audio and this high-level conceptual space. During fine-tuning, Bagpiper adopts a caption-then-process workflow, simulating an intermediate cognitive reasoning step to solve diverse tasks without knowing prior task-specific practice. Experimentally, Bagpiper achieves universal generation that can uniformly generate speech, sound effects, music, and their arbitrary combinations. It also maintains comparable performance with the 7B Qwen-2.5-Omni for audio understanding. To the best of our knowledge, Bagpiper is among the first works that achieve open-ended audio understanding and generation on speech, sound, and music. Model, data, and code will be released at Bagpiper Home Page.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。