arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.10667cs.SDeess.AS
收藏

音频文本模型尚未利用自然语言

Audio-Text Models Do Not Yet Leverage Natural Language

Adobe Research(奥多比研究院)

Ho-Hsiang Wu, Oriol Nieto, Juan Pablo Bello, Justin Salamon

摘要:

多模态对比学习技术在音频文本领域的研究已迅速成为一个高度活跃的领域。大多数作品都是用标准的音频检索和分类基准进行评估的,假设(i)这些模型能够利用自然语言中包含的丰富信息,以及(ii)当前的基准能够捕捉到这些信息的细微差别。在这项工作中,我们表明,国家的最先进的音频文本模型还没有真正理解自然语言,特别是上下文的概念,如顺序或并发排序的声音事件。我们的研究结果表明,现有的基准不足以评估这些模型的能力,以匹配复杂的背景下,从音频和文本形式。我们提出了一个基于transformer的架构,并表明,与以前的工作,它是能够建模的文本和音频中的声音事件之间的顺序关系,给出适当的基准数据。我们提倡收集或生成额外的、多样化的数据,以使未来的研究能够充分利用自然语言进行音频文本建模。

英文摘要:

Multi-modal contrastive learning techniques in the audio-text domain have quickly become a highly active area of research. Most works are evaluated with standard audio retrieval and classification benchmarks assuming that (i) these models are capable of leveraging the rich information contained in natural language, and (ii) current benchmarks are able to capture the nuances of such information. In this work, we show that state-of-the-art audio-text models do not yet really understand natural language, especially contextual concepts such as sequential or concurrent ordering of sound events. Our results suggest that existing benchmarks are not sufficient to assess these models' capabilities to match complex contexts from the audio and text modalities. We propose a Transformer-based architecture and show that, unlike prior work, it is capable of modeling the sequential relationship between sound events in the text and audio, given appropriate benchmark data. We advocate for the collection or generation of additional, diverse, data to allow future research to fully leverage natural language for audio-text modeling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑