CALM：对比对齐的音频语言多速率和多模态表示
CALM: Contrastive Aligned Audio-Language Multirate and Multimodal Representations
摘要：
获得音频和词汇输入的多模态表示是自然语言理解（NLU）的核心问题。在本文中，我们提出了对比对齐的音频语言多速率和多模态表示（CALM），一种使用音频和词汇输入中固有的对比和多速率信息来学习多模态表示的方法。该模型将声学和词汇信息对齐在预训练的仅语言上下文嵌入模型的输入嵌入空间中。通过将音频表示与预训练的语言表示对齐，并利用声学输入之间的对比信息，CALM能够在短短几个小时的训练时间内引导音频嵌入，与现有的音频表示模型竞争。在操作上，通过频谱Transformer（SpecTran）使用线性化补丁来处理音频频谱图，频谱transformer（SpecTran）使用对比音频语言预训练目标来训练，以将来自类似查询的音频和语言对齐。随后，所导出的声学和词汇标记表示被输入到多模态Transformer中以并入话语级上下文并导出所提出的CALM表示。我们表明，这些预训练的嵌入随后可以用于多模态监督任务，并证明了所提出的预训练步骤在两个嵌入空间的对齐和预训练的多速率性质方面的好处。我们的系统表现出10- 25%的改进，现有的情感识别系统，包括国家的最先进的三模态系统在各种评价目标。
英文摘要：
Deriving multimodal representations of audio and lexical inputs is a central problem in Natural Language Understanding (NLU). In this paper, we present Contrastive Aligned Audio-Language Multirate and Multimodal Representations (CALM), an approach for learning multimodal representations using contrastive and multirate information inherent in audio and lexical inputs. The proposed model aligns acoustic and lexical information in the input embedding space of a pretrained language-only contextual embedding model. By aligning audio representations to pretrained language representations and utilizing contrastive information between acoustic inputs, CALM is able to bootstrap audio embedding competitive with existing audio representation models in only a few hours of training time. Operationally, audio spectrograms are processed using linearized patches through a Spectral Transformer (SpecTran) which is trained using a Contrastive Audio-Language Pretraining objective to align audio and language from similar queries. Subsequently, the derived acoustic and lexical tokens representations are input into a multimodal transformer to incorporate utterance level context and derive the proposed CALM representations. We show that these pretrained embeddings can subsequently be used in multimodal supervised tasks and demonstrate the benefits of the proposed pretraining steps in terms of the alignment of the two embedding spaces and the multirate nature of the pretraining. Our system shows 10-25\% improvement over existing emotion recognition systems including state-of-the-art three-modality systems under various evaluation objectives.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
