arXiv 2303.176112023-03-29cs.HCcs.AIcs.SDeess.AS

基于Transformer的自监督多模态表征学习用于可穿戴情感识别

Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition

Univ. Lille（里尔大学）；CNRS（法国国家科学研究中心）；Centrale Lille（里尔中央理工学院）；Institut Mines-Télécom（法国国立高等矿业电信学校联盟）；IMT Nord Europe（法国国立高等矿业电信学校联盟北欧分校）；INSERM（法国国家健康与医学研究院）；University of Lille（里尔大学）

Yujin Wu, Mohamed Daoudi, Ali Amad

备注：: Accepted IEEE Transactions On Affective Computing

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对可穿戴情感识别中多模态数据融合与标注数据有限导致过拟合的问题，提出一种基于Transformer的自监督学习框架，通过模态特定编码器与共享编码器实现高效多模态融合，并在下游情感分类任务中取得最优性能。

摘要：

近年来，基于外周生理信号的可穿戴情感识别因其低侵入性以及在现实生活场景中的适用性而受到广泛关注。然而，如何有效融合多模态数据仍然是一个具有挑战性的问题。此外，在标注数据有限的情况下，传统的全监督方法容易出现过拟合。为了解决上述问题，我们提出了一种用于可穿戴情感识别的新型自监督学习（SSL）框架，该框架通过基于时间卷积的模态特定编码器和基于Transformer的共享编码器实现了高效的多模态融合，能够同时捕获模态内和模态间的相关性。大量无标签数据通过五种信号变换自动分配标签，所提出的SSL模型以信号变换识别作为前置任务进行预训练，从而能够提取与情感相关的下游任务的通用多模态表征。在评估方面，所提出的SSL模型首先在一个大规模自采集的生理数据集上进行预训练，随后将得到的编码器在三个公开的监督情感识别数据集上进行冻结或微调。最终，我们基于SSL的方法在各种情感分类任务中取得了最先进的结果。同时，与全监督方法相比，所提出的模型在低数据量情况下被证明更加准确和鲁棒。

英文摘要：

Recently, wearable emotion recognition based on peripheral physiological signals has drawn massive attention due to its less invasive nature and its applicability in real-life scenarios. However, how to effectively fuse multimodal data remains a challenging problem. Moreover, traditional fully-supervised based approaches suffer from overfitting given limited labeled data. To address the above issues, we propose a novel self-supervised learning (SSL) framework for wearable emotion recognition, where efficient multimodal fusion is realized with temporal convolution-based modality-specific encoders and a transformer-based shared encoder, capturing both intra-modal and inter-modal correlations. Extensive unlabeled data is automatically assigned labels by five signal transforms, and the proposed SSL model is pre-trained with signal transformation recognition as a pretext task, allowing the extraction of generalized multimodal representations for emotion-related downstream tasks. For evaluation, the proposed SSL model was first pre-trained on a large-scale self-collected physiological dataset and the resulting encoder was subsequently frozen or fine-tuned on three public supervised emotion recognition datasets. Ultimately, our SSL-based method achieved state-of-the-art results in various emotion classification tasks. Meanwhile, the proposed model proved to be more accurate and robust compared to fully-supervised methods on low data regimes.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。