使声音和视觉线索变得重要：CH-SIMS v2 .0数据集和AV混淆一致性模块
Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module
摘要：
多模态情感分析（Multimodalsentimentanalysis，MSA）是一个新兴的研究领域，它旨在通过结合听觉和视觉模态来改进基于文本的情感分析，在人机交互（Human-ComputerInteraction，HCI）中具有潜在的应用价值。然而，现有的研究发现，听觉和视觉通道的贡献远远小于文本通道，被称为文本主导。在这种情况下，在这项工作中，我们强调非语言线索的事项MSA任务。首先，从资源的角度来看，我们提出了CH-SIMS v2.0数据集，CH-SIMS的扩展和增强。与原始数据集相比，CH-SIMS v2.0的大小增加了一倍，其中包含另外2121个具有单峰和多峰注释的细化视频片段，并收集了10161个未标记的原始视频片段，这些片段具有丰富的声学和视觉情感背景，以突出非语言线索用于情感预测。其次，从模型的角度出发，利用CH-SIMS v2. 0中的单峰注释和无监督数据，提出了声学视觉混淆一致性（AV-MC）框架。所设计的模态混合模块可以看作是一个增强，它混合了来自不同视频的声学和视觉模态。通过将未观察到的多模态上下文与文本一起绘制，模型可以学习了解不同的非语言上下文以进行情感预测。我们的评估表明，CH-SIMS v2.0和AV-MC框架都可以进一步研究发现承载情感的声学和视觉线索，并为现实世界场景中可解释的端到端HCI应用铺平道路。
英文摘要：
Multimodal sentiment analysis (MSA), which supposes to improve text-based sentiment analysis with associated acoustic and visual modalities, is an emerging research area due to its potential applications in Human-Computer Interaction (HCI). However, the existing researches observe that the acoustic and visual modalities contribute much less than the textual modality, termed as text-predominant. Under such circumstances, in this work, we emphasize making non-verbal cues matter for the MSA task. Firstly, from the resource perspective, we present the CH-SIMS v2.0 dataset, an extension and enhancement of the CH-SIMS. Compared with the original dataset, the CH-SIMS v2.0 doubles its size with another 2121 refined video segments with both unimodal and multimodal annotations and collects 10161 unlabelled raw video segments with rich acoustic and visual emotion-bearing context to highlight non-verbal cues for sentiment prediction. Secondly, from the model perspective, benefiting from the unimodal annotations and the unsupervised data in the CH-SIMS v2.0, the Acoustic Visual Mixup Consistent (AV-MC) framework is proposed. The designed modality mixup module can be regarded as an augmentation, which mixes the acoustic and visual modalities from different videos. Through drawing unobserved multimodal context along with the text, the model can learn to be aware of different non-verbal contexts for sentiment prediction. Our evaluations demonstrate that both CH-SIMS v2.0 and AV-MC framework enables further research for discovering emotion-bearing acoustic and visual cues and paves the path to interpretable end-to-end HCI applications for real-world scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
