用于视听语音情感识别的向量量化掩码自编码器
A vector quantized masked autoencoder for audiovisual speech emotion recognition
AI总结：
本文提出VQ-MAE-AV自监督多模态模型，通过向量量化与掩码自编码器从无标签视听语音中学习帧级和序列级表示，并在多个数据集上取得最先进的情感识别性能。
摘要：
情感识别中的一个重要挑战是开发能够利用无标签训练数据的方法。本文提出了VQ-MAE-AV模型，这是一种自监督多模态模型，利用掩码自编码器在无标签条件下学习视听语音的表示。该模型包含向量量化变分自编码器，将原始音频和视觉语音数据压缩为离散token。视听语音token用于训练一个多模态掩码自编码器，该自编码器由带注意力机制的编码器-解码器架构组成。该模型旨在提取视听语音的局部（即帧级）和全局（即序列级）表示。在自监督预训练阶段，VQ-MAE-AV模型在大规模无标签视听语音数据集上进行训练，任务是重建随机掩码的视听语音token，并结合对比学习策略。在此预训练过程中，编码器学习提取视听语音的表示，该表示可后续用于情感识别。在有监督微调阶段，在VQ-MAE-AV编码器之上训练一个小型分类模型用于情感识别任务。所提出的方法在受控条件和真实场景条件下的多个数据集上均取得了最先进的情感识别结果。
英文摘要：
An important challenge in emotion recognition is to develop methods that can leverage unlabeled training data. In this paper, we propose the VQ-MAE-AV model, a self-supervised multimodal model that leverages masked autoencoders to learn representations of audiovisual speech without labels. The model includes vector quantized variational autoencoders that compress raw audio and visual speech data into discrete tokens. The audiovisual speech tokens are used to train a multimodal masked autoencoder that consists of an encoder-decoder architecture with attention mechanisms. The model is designed to extract both local (i.e., at the frame level) and global (i.e., at the sequence level) representations of audiovisual speech. During self-supervised pre-training, the VQ-MAE-AV model is trained on a large-scale unlabeled dataset of audiovisual speech, for the task of reconstructing randomly masked audiovisual speech tokens and with a contrastive learning strategy. During this pre-training, the encoder learns to extract a representation of audiovisual speech that can be subsequently leveraged for emotion recognition. During the supervised fine-tuning stage, a small classification model is trained on top of the VQ-MAE-AV encoder for an emotion recognition task. The proposed approach achieves state-of-the-art emotion recognition results across several datasets in both controlled and in-the-wild conditions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
