This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.
Stock Movement Prediction with Multimodal Stable Fusion via Gated Cross-Attention Mechanism
School of Information and Electronic Engineering, Zhejiang University of Science and Technology（浙江理工大学信息与电子工程学院）；Department of Finance Accounting and Economics, Business School of Nottingham University（诺丁汉大学商学院金融会计与经济学系）
MMLatch: Bottom-up Top-down Fusion for Multimodal Sentiment Analysis
National Technical University of Athens（雅典国家技术大学）；Institute for Language and Speech Processing, Athena Research Center（雅典娜研究中心语言与语音处理研究所）；Behavioral Signal Technologies（行为信号技术公司）
Variational Fusion for Multimodal Sentiment Analysis
A*STAR AI Initiative, Institute of High Performance Computing（A*STAR人工智能计划、高性能计算研究院）；Adobe Research（奥多比研究院）；University of Michigan（密歇根大学）；Centro de Investigación en Computación, Instituto Politécnico Nacional（国立理工学院计算研究中心）；SUTD（新加坡科技设计大学）
