HiCMAE：用于自监督视听情感识别的分层对比掩码自编码器
HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
AI总结：
本文提出自监督框架HiCMAE，通过掩码数据建模、对比学习及三管齐下的分层特征学习策略，在9个数据集上显著超越现有监督与自监督视听情感识别方法。
摘要：
视听情感识别（AVER）近年来因其在构建情感感知智能机器中的关键作用而受到越来越多的关注。该领域以往的工作主要由监督学习范式主导。尽管取得了显著进展，但由于AVER中长期存在的数据稀缺问题，监督学习正面临瓶颈。受自监督学习最新进展的启发，我们提出了分层对比掩码自编码器（HiCMAE），这是一种新颖的自监督框架，利用大规模无标注视听数据上的大规模自监督预训练来推动AVER的发展。遵循自监督视听表示学习的先前工作，HiCMAE采用两种主要的自监督形式进行预训练，即掩码数据建模和对比学习。与它们仅关注顶层表示而忽视中间层显式指导的做法不同，HiCMAE开发了一种三管齐下的策略，以促进分层视听特征学习并提高所学表示的整体质量。为了验证HiCMAE的有效性，我们在9个数据集上进行了广泛的实验，涵盖分类和维度AVER任务。实验结果表明，我们的方法显著优于最先进的监督和自监督视听方法，这表明HiCMAE是一个强大的视听情感表示学习器。代码和模型将在https://github.com/sunlicai/HiCMAE上公开提供。
英文摘要：
Audio-Visual Emotion Recognition (AVER) has garnered increasing attention in recent years for its critical role in creating emotion-ware intelligent machines. Previous efforts in this area are dominated by the supervised learning paradigm. Despite significant progress, supervised learning is meeting its bottleneck due to the longstanding data scarcity issue in AVER. Motivated by recent advances in self-supervised learning, we propose Hierarchical Contrastive Masked Autoencoder (HiCMAE), a novel self-supervised framework that leverages large-scale self-supervised pre-training on vast unlabeled audio-visual data to promote the advancement of AVER. Following prior arts in self-supervised audio-visual representation learning, HiCMAE adopts two primary forms of self-supervision for pre-training, namely masked data modeling and contrastive learning. Unlike them which focus exclusively on top-layer representations while neglecting explicit guidance of intermediate layers, HiCMAE develops a three-pronged strategy to foster hierarchical audio-visual feature learning and improve the overall quality of learned representations. To verify the effectiveness of HiCMAE, we conduct extensive experiments on 9 datasets covering both categorical and dimensional AVER tasks. Experimental results show that our method significantly outperforms state-of-the-art supervised and self-supervised audio-visual methods, which indicates that HiCMAE is a powerful audio-visual emotion representation learner. Codes and models will be publicly available at https://github.com/sunlicai/HiCMAE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
