面向多模态情感分析的三重解耦表示学习
Triple Disentangled Representation Learning for Multimodal Affective Analysis
AI总结:
针对多模态情感分析中模态特定表示含无关或冲突信息的问题,提出 TriDiRA 三重解耦方法,仅融合模态不变与有效模态特定表示,并在四个基准上超越 SOTA。
摘要:
多模态学习凭借各模态的全面信息(尤其是互补信息),在情感分析任务中展现出显著优势。因此,许多新兴研究聚焦于从输入数据中解耦模态不变表示和模态特定表示,随后将其融合以进行预测。然而,我们的研究表明,模态特定表示可能包含与任务无关或冲突的信息,这会降低所学多模态表示的有效性。我们重新审视了解耦问题,并提出了一种新颖的三重解耦方法 TriDiRA,该方法从输入数据中解耦出模态不变、有效模态特定和无效模态特定表示。通过仅融合模态不变表示和有效模态特定表示,TriDiRA 能够在模型训练期间显著缓解跨模态无关信息和冲突信息的影响。在四个基准数据集上开展的大量实验证明了我们的三重解耦方法的有效性和泛化能力,其性能优于 SOTA 方法。
英文摘要:
Multimodal learning has exhibited a significant advantage in affective analysis tasks owing to the comprehensive information of various modalities, particularly the complementary information. Thus, many emerging studies focus on disentangling the modality-invariant and modality-specific representations from input data and then fusing them for prediction. However, our study shows that modality-specific representations may contain information that is irrelevant or conflicting with the tasks, which downgrades the effectiveness of learned multimodal representations. We revisit the disentanglement issue, and propose a novel triple disentanglement approach, TriDiRA, which disentangles the modality-invariant, effective modality-specific and ineffective modality-specific representations from input data. By fusing only the modality-invariant and effective modality-specific representations, TriDiRA can significantly alleviate the impact of irrelevant and conflicting information across modalities during model training. Extensive experiments conducted on four benchmark datasets demonstrate the effectiveness and generalization of our triple disentanglement, which outperforms SOTA methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。