arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2011.06102cs.AI
收藏

通过模态预训练和注意力提高多模态准确性

Improving Multimodal Accuracy Through Modality Pre-training and Attention

University of Maryland(马里兰大学);Comcast Labs(康卡斯特实验室)

Aya Abdelsalam Ismail, Mahmudul Hasan, Faisal Ishtiaq

摘要:

训练多模态网络是一项挑战,它需要复杂的架构来实现合理的性能。我们发现,这种现象的原因之一是不同的方式之间的收敛速度的差异。我们通过在整个网络的端到端训练之前独立地在多模态架构中预训练特定于模态的子网络来解决这个问题。此外,我们表明,在预训练后的子网络之间添加注意力机制有助于在模糊场景中识别最重要的模态,从而提高性能。我们证明,通过执行这两个技巧,一个简单的网络可以实现类似的性能,一个复杂的架构,是显着更昂贵的训练的多个任务,包括情感分析,情感识别和说话人特质识别。

英文摘要:

Training a multimodal network is challenging and it requires complex architectures to achieve reasonable performance. We show that one reason for this phenomena is the difference between the convergence rate of various modalities. We address this by pre-training modality-specific sub-networks in multimodal architectures independently before end-to-end training of the entire network. Furthermore, we show that the addition of an attention mechanism between sub-networks after pre-training helps identify the most important modality during ambiguous scenarios boosting the performance. We demonstrate that by performing these two tricks a simple network can achieve similar performance to a complicated architecture that is significantly more expensive to train on multiple tasks including sentiment analysis, emotion recognition, and speaker trait recognition.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑