arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.07203cs.SDeess.AS
收藏

ParaCLAP——面向计算副语言任务的通用语言-音频模型

ParaCLAP -- Towards a general language-audio model for computational paralinguistic tasks

University of Augsburg(奥格斯堡大学);Technical University of Munich(慕尼黑工业大学);Imperial College London(伦敦帝国学院)

Xin Jing, Andreas Triantafyllopoulos, Björn Schuller

AI总结:

本文提出面向计算副语言任务的 ParaCLAP 模型,通过新颖的音频-语言查询构建流程弥补 CP 领域配对数据缺失,并在多个 CP 任务上超越开源最先进模型。

摘要:

对比语言-音频预训练(CLAP)近来已成为一种使音频分析更具泛化能力的方法。具体而言,CLAP 风格的模型能够“回答”多样化的语言查询,从而将音频模型的能力扩展到封闭标签集之外。然而,CLAP 依赖大量(音频,查询)对进行预训练。虽然此类数据集在通用音频任务(如字幕生成或声音事件检测)中可用,但针对计算副语言(CP)任务,目前尚不存在音频与文本查询相匹配的数据集。因此,研究社区只能依赖为通用音频训练的通用 CLAP 模型,效果有限。在本研究中,我们探索了 ParaCLAP 的训练考量,ParaCLAP 是一种适用于 CP 的 CLAP 风格模型,其中包含一种用于创建音频-语言查询的新颖流程。我们在一组计算副语言任务上展示了其有效性,结果表明它超越了开源最先进模型的性能。

英文摘要:

Contrastive language-audio pretraining (CLAP) has recently emerged as a method for making audio analysis more generalisable. Specifically, CLAP-style models are able to `answer' a diverse set of language queries, extending the capabilities of audio models beyond a closed set of labels. However, CLAP relies on a large set of (audio, query) pairs for pretraining. While such sets are available for general audio tasks, like captioning or sound event detection, there are no datasets with matched audio and text queries for computational paralinguistic (CP) tasks. As a result, the community relies on generic CLAP models trained for general audio with limited success. In the present study, we explore training considerations for ParaCLAP, a CLAP-style model suited to CP, including a novel process for creating audio-language queries. We demonstrate its effectiveness on a set of computational paralinguistic tasks, where it is shown to surpass the performance of open-source state-of-the-art models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑