arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2611.04480cs.CLcs.SDeess.AS
收藏

ReListen:使用冻结音频语言模型进行无需训练的时长一小时音频中的证据定位

ReListen: Training-Free Evidence Localization in Hour-Long Audio with Frozen Audio Language Models

School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);Microsoft CoreAI(微软核心人工智能部门);Microsoft Research Asia(微软亚洲研究院)

Ming-Hao Hsu, Shujie Liu, Yuxuan Hu, Yuxiang Wang, Yingda Shen, Jinyu Li, Zhizheng Wu

AI总结:

ReListen是无需训练的方法,用冻结音频语言模型通过Jev风格扫描定位一小时音频中的证据,仅用少量音频即可在VoiceGiraffe等数据集上实现优于转录的问答表现,且速度更快、内存占用稳定。

摘要:

音频语言模型旨在基于音频本身回答问题,但一小时的录音通常无法适配其上下文,且答案往往取决于其中的几秒内容。先转录录音虽能适配上下文,但会丢失声音和语音信息;而需返回音频的系统则需针对该行为进行训练。我们提出ReListen,一种无需训练的方法,其中冻结音频语言模型在回答前自行查找证据。采用Jev风格的扫描方式,模型对每个20秒窗口进行评分,评分依据为该窗口有助于回答问题的概率,随后从得分最高的一分钟内容中给出答案。对于VoiceGiraffe数据集的问题,该一分钟内容与标注证据的重叠率远高于转录检索,达88.5%。无需任何修改,相同的扫描方式也可将LAT-Bench上的事件定位与专门针对时间定位训练的模型表现相当。基于此一分钟内容给出的答案,仅使用音频的一小部分,其表现与聆听完整录音相当或更优;在VoiceGiraffe数据集上,其表现比聆听完整录音的转录结果高出9.2个百分点。所选证据更适合聆听而非阅读:对于相同的片段,阅读其转录内容在语音内容相关问题上的表现与聆听相当,但在关于声音和语音的问题上,聆听明显更准确。此外,ReListen的回答速度快于所有基于转录的系统,且即使是无法在单个GPU上完整聆听的长录音,其内存占用也保持恒定。因此,冻结音频语言模型可自行搜索时长一小时的录音,无需转录或额外训练。

英文摘要:

Audio language models are meant to answer questions from the audio itself, but an hour-long recording rarely fits in their context, and the answer often depends on a few seconds of it. Transcribing the recording first makes it fit but discards sounds and voices, while systems that return to the audio must be trained for this behavior. We propose ReListen, a training-free method in which a frozen audio language model finds its own evidence before answering. In a Jev-style scan, the model scores each 20-second window by the probability that it helps answer the question, and it then answers from the highest-scoring minute. This minute overlaps the annotated evidence far more often than transcript retrieval, for 88.5% of VoiceGiraffe questions. Without any change, the same scan also grounds events on LAT-Bench as well as a model trained for temporal grounding. Answering from it matches or exceeds hearing the whole recording with a small fraction of the audio, and on VoiceGiraffe it exceeds a transcript of the whole recording by 9.2 points. The selected evidence is also worth hearing rather than reading. Given the same excerpts, reading their transcript matches listening on speech content, but listening is clearly more accurate on questions about sounds and voices. Moreover, ReListen answers faster than every transcript-based system, and its memory stays constant even for recordings too long to hear whole on one GPU. Therefore, a frozen audio language model can search hour-long recordings by itself, without a transcript or additional training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑