arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.13669cs.AI
收藏

相关控制多模态情感分析中的双-双模态融合

Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis

SUTD(新加坡科技设计大学);IPN(墨西哥国立理工学院);CMU(卡内基梅隆大学)

Wei Han, Hui Chen, Alexander Gelbukh, Amir Zadeh, Louis-philippe Morency, Soujanya Poria

摘要:

多模态情感分析旨在提取和整合从多模态数据中收集的语义信息,以识别多模态数据中表达的情感和情感。这一研究领域的主要关注点在于开发一个非凡的融合方案,可以提取和整合各种模态的关键信息。然而,一个问题,可能会限制以前的工作,以达到更高的水平是缺乏适当的建模模式之间的独立性和相关性之间的竞争的动态,这可能会恶化融合的结果,导致崩溃的模态特定的特征空间或引入额外的噪声。为了缓解这一问题,我们提出了双双峰融合网络(BBFN),这是一种新型的端到端网络,可以对成对的模态表示进行融合(相关性增量)和分离(差异增量)。这两个部分同时训练,以便模拟它们之间的战斗。该模型采用两个双峰对作为输入,由于已知的信息不平衡的模态。此外,我们利用Transformer架构中的门控控制机制来进一步改善最终输出。在CMU-MOSI、CMU-MOSEI和UR-FUNNY三个数据集上的实验结果表明,该模型的性能明显优于SOTA。这项工作的实施情况可在https://github.com/declare-lab/multimodal-deep-learning上查阅。

英文摘要:

Multimodal sentiment analysis aims to extract and integrate semantic information collected from multiple modalities to recognize the expressed emotions and sentiment in multimodal data. This research area's major concern lies in developing an extraordinary fusion scheme that can extract and integrate key information from various modalities. However, one issue that may restrict previous work to achieve a higher level is the lack of proper modeling for the dynamics of the competition between the independence and relevance among modalities, which could deteriorate fusion outcomes by causing the collapse of modality-specific feature space or introducing extra noise. To mitigate this, we propose the Bi-Bimodal Fusion Network (BBFN), a novel end-to-end network that performs fusion (relevance increment) and separation (difference increment) on pairwise modality representations. The two parts are trained simultaneously such that the combat between them is simulated. The model takes two bimodal pairs as input due to the known information imbalance among modalities. In addition, we leverage a gated control mechanism in the Transformer architecture to further improve the final output. Experimental results on three datasets (CMU-MOSI, CMU-MOSEI, and UR-FUNNY) verifies that our model significantly outperforms the SOTA. The implementation of this work is available at https://github.com/declare-lab/multimodal-deep-learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑