一种面向语义理解、具有跨模态关系与层次交互注意力的多模态多任务框架
A Multimodal-Multitask Framework with Cross-modal Relation and Hierarchical Interactive Attention for Semantic Comprehension
AI总结：
本文提出多模态多任务框架 MM-ORIENT，通过跨模态关系图和层次交互单模态注意力在无显式模态交互下降低噪声影响，并保留单模态判别信息，在三个数据集上验证了其多任务语义理解效果。
摘要：
多模态学习中的一个主要挑战是单个模态内部存在噪声。这种噪声本质上会影响最终得到的多模态表示，尤其是当这些表示是通过不同模态之间的显式交互获得时。此外，多模态融合技术在追求强联合表示的同时，可能会忽略单个模态内部有价值的判别性信息。为此，我们提出了一种多模态多任务框架 MM-ORIENT，该框架利用跨模态关系（cross-modal relation）和层次交互注意力（hierarchical interactive attention），对多个任务均有效。所提方法在不同模态之间不进行显式交互的情况下，以跨模态方式获取多模态表示，从而在潜在阶段降低噪声影响。为实现这一目标，我们提出了跨模态关系图，通过重构单模态特征来获得多模态表示。特征基于节点邻域进行重构，而邻域由另一模态的特征决定。我们还提出了层次交互单模态注意力（HIMA），用于关注模态内部的相关信息。跨模态关系图有助于理解两个模态之间的高阶关系，而 HIMA 通过在后期融合之前学习单个模态的判别性特征来辅助多任务学习。最后，在三个数据集上的广泛实验评估表明，所提方法能够有效地理解多模态内容并完成多个任务。
英文摘要：
A major challenge in multimodal learning is the presence of noise within individual modalities. This noise inherently affects the resulting multimodal representations, especially when these representations are obtained through explicit interactions between different modalities. Moreover, the multimodal fusion techniques while aiming to achieve a strong joint representation, can neglect valuable discriminative information within the individual modalities. To this end, we propose a Multimodal-Multitask framework with crOss-modal Relation and hIErarchical iNteractive aTtention (MM-ORIENT) that is effective for multiple tasks. The proposed approach acquires multimodal representations cross-modally without explicit interaction between different modalities, reducing the noise effect at the latent stage. To achieve this, we propose cross-modal relation graphs that reconstruct monomodal features to acquire multimodal representations. The features are reconstructed based on the node neighborhood, where the neighborhood is decided by the features of a different modality. We also propose Hierarchical Interactive Monomadal Attention (HIMA) to focus on pertinent information within a modality. While cross-modal relation graphs help comprehend high-order relationships between two modalities, HIMA helps in multitasking by learning discriminative features of individual modalities before late-fusing them. Finally, extensive experimental evaluation on three datasets demonstrates that the proposed approach effectively comprehends multimodal content for multiple tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
