TAC:带时间戳的音频描述
TAC: Timestamped Audio Captioning
AI总结:
TAC通过生成时间接地的音频描述提升复杂声音场景的理解,结合TAC-V实现音频-视觉语义桥梁,提升文本推理性能。
摘要:
大型音频语言模型在处理复杂声音场景中的重叠事件时存在困难,导致时间不一致的描述和频繁的幻觉。我们介绍了带时间戳的音频描述器(TAC),该模型能够生成不同程度细节和分辨率的时间接地音频描述。TAC通过合成数据管道训练,从真实世界音频源中构建具有挑战性和动态性的混合物,从而在现实多音条件下的鲁棒学习。在事件检测和密集描述方面,TAC优于所有竞争方法,具有低幻觉率和准确的时间接地。我们还引入了TAC-V,一种音频-视觉管道,用于生成语义丰富的音频-视觉描述。然后我们展示TAC和TAC-V作为纯文本推理器的“语义桥梁”:简单的TAC→LLM和TAC-V→LLM级联在音频(MMAU-Pro,MMSU,MMAR)和音频-视觉(DailyOmni,VideoHolmes)理解和推理基准上分别达到了最先进的分数。
英文摘要:
Large Audio Language Models struggle to disentangle overlapping events in complex acoustic scenes, yielding temporally inconsistent captions and frequent hallucinations. We introduce Timestamped Audio Captioner (TAC), a model that produces temporally grounded audio descriptions at varying degrees of detail and resolution. TAC is trained with a synthetic data pipeline that constructs challenging and dynamic mixtures from real-world audio sources, enabling robust learning under realistic polyphonic conditions. Across event detection and dense captioning, TAC outperforms all competing methods, with a low hallucination rate and accurate temporal grounding. We also introduce TAC-V, an audio-visual pipeline to generate semantically rich audio-visual descriptions. We then show that TAC and TAC-V serves as a "semantic bridge" for a text-only reasoner: a simple TAC$\rightarrow$LLM and TAC-V$\rightarrow$LLM cascade achieves state-of-the-art scores on benchmarks for both audio (MMAU-Pro, MMSU, MMAR) and audio-visual (DailyOmni, VideoHolmes) understanding and reasoning respectively.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。