arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.15766cs.SD
收藏

TAC:带时间戳的音频描述

TAC: Timestamped Audio Captioning

University of Maryland, College Park, USA(美国马里兰大学 College Park 分校);Adobe Research, USA(Adobe 研究院);OpenAI, USA (work done while at Adobe)(OpenAI, USA)

Sonal Kumar, Prem Seetharaman, Ke Chen, Oriol Nieto, Jiaqi Su, Zhepei Wang, Rithesh Kumar, Dinesh Manocha, Nicholas J. Bryan, Zeyu Jin, Justin Salamon

AI总结:

TAC通过生成时间接地的音频描述提升复杂声音场景的理解,结合TAC-V实现音频-视觉语义桥梁,提升文本推理性能。

摘要:

大型音频语言模型在处理复杂声音场景中的重叠事件时存在困难,导致时间不一致的描述和频繁的幻觉。我们介绍了带时间戳的音频描述器(TAC),该模型能够生成不同程度细节和分辨率的时间接地音频描述。TAC通过合成数据管道训练,从真实世界音频源中构建具有挑战性和动态性的混合物,从而在现实多音条件下的鲁棒学习。在事件检测和密集描述方面,TAC优于所有竞争方法,具有低幻觉率和准确的时间接地。我们还引入了TAC-V,一种音频-视觉管道,用于生成语义丰富的音频-视觉描述。然后我们展示TAC和TAC-V作为纯文本推理器的“语义桥梁”:简单的TAC→LLM和TAC-V→LLM级联在音频(MMAU-Pro,MMSU,MMAR)和音频-视觉(DailyOmni,VideoHolmes)理解和推理基准上分别达到了最先进的分数。

英文摘要:

Large Audio Language Models struggle to disentangle overlapping events in complex acoustic scenes, yielding temporally inconsistent captions and frequent hallucinations. We introduce Timestamped Audio Captioner (TAC), a model that produces temporally grounded audio descriptions at varying degrees of detail and resolution. TAC is trained with a synthetic data pipeline that constructs challenging and dynamic mixtures from real-world audio sources, enabling robust learning under realistic polyphonic conditions. Across event detection and dense captioning, TAC outperforms all competing methods, with a low hallucination rate and accurate temporal grounding. We also introduce TAC-V, an audio-visual pipeline to generate semantically rich audio-visual descriptions. We then show that TAC and TAC-V serves as a "semantic bridge" for a text-only reasoner: a simple TAC$\rightarrow$LLM and TAC-V$\rightarrow$LLM cascade achieves state-of-the-art scores on benchmarks for both audio (MMAU-Pro, MMSU, MMAR) and audio-visual (DailyOmni, VideoHolmes) understanding and reasoning respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑