Multimodal learning mimics the reasoning process of the human multi-sensory system, which is used to perceive the surrounding world. While making a prediction, the human brain tends to relate crucial cues from multiple sources of information. In this work, we propose a novel multimodal fusion module that learns to emphasize more contributive features across all modalities. Specifically, the proposed Multimodal Split Attention Fusion (MSAF) module splits each modality into channel-wise equal feature blocks and creates a joint representation that is used to generate soft attention for each channel across the feature blocks. Further, the MSAF module is designed to be compatible with features of various spatial dimensions and sequence lengths, suitable for both CNNs and RNNs. Thus, MSAF can be easily added to fuse features of any unimodal networks and utilize existing pretrained unimodal model weights. To demonstrate the effectiveness of our fusion module, we design three multimodal networks with MSAF for emotion recognition, sentiment analysis, and action recognition tasks. Our approach achieves competitive results in each task and outperforms other application-specific networks and multimodal fusion benchmarks.
A Multi-Stage Adaptive Feature Fusion Neural Network for Multimodal Gait Recognition
Central South University（中南大学）；Southern University of Science and Technology（南方科技大学）；Research Institute of Trustworthy Autonomous System, Southern University of Science and Technology（南方科技大学可信自主系统研究院）
We present a simple, yet effective and flexible method for action recognition supporting multiple sensor modalities. Multivariate signal sequences are encoded in an image and are then classified using a recently proposed EfficientNet CNN architecture. Our focus was to find an approach that generalizes well across different sensor modalities without specific adaptions while still achieving good results. We apply our method to 4 action recognition datasets containing skeleton sequences, inertial and motion capturing measurements as well as \wifi fingerprints that range up to 120 action classes. …
Question answering biases in video QA datasets can mislead multimodal model to overfit to QA artifacts and jeopardize the model's ability to generalize. Understanding how strong these QA biases are and where they come from helps the community measure progress more accurately and provide researchers insights to debug their models. In this paper, we analyze QA biases in popular video question answering datasets and discover pretrained language models can answer 37-48% questions correctly without using any multimodal context information, far exceeding the 20% random guess baseline for 5-choose-1 …
GMoT: Gated Motion-Aware Tokenization for Fine-Grained Micro-Gesture Video Reasoning with Multimodal LLMs
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Great Bay University（大湾区大学）；Dongguan Key Laboratory for Intelligence and Information Technology（东莞市智能信息技术重点实验室）
Reliability-Aware Multimodal Sentiment Prediction Under Local Modality Missingness
Department of Network Intelligence, Pengcheng Laboratory（鹏城实验室网络智能部）；School of Mathematics, Sun Yat-sen University（中山大学数学学院）；School of Physics and Astronomy, Sun Yat-sen University（中山大学物理与天文学院）
Tsinghua University（清华大学）；Toutiao AI Lab（今日头条人工智能实验室）；UESTC（电子科技大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
In this work we formulate the problem of image captioning as a multimodal translation task. Analogous to machine translation, we present a sequence-to-sequence recurrent neural networks (RNN) model for image caption generation. Different from most existing work where the whole image is represented by convolutional neural network (CNN) feature, we propose to represent the input image as a sequence of detected objects which feeds as the source sequence of the RNN model. In this way, the sequential representation of an image can be naturally translated to a sequence of words, as the target sequen…
Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognition
University of the Basque Country UPV/EHU（巴斯克大学UPV/EHU）；Laboratory of LESIA, University of Biskra（贝斯克拉大学LESIA实验室）；Lab. de Génie Electrique (LAGE), University Kasdi Merbah Ouargla（奥尔加拉大学LAGE实验室）；Institute of Electronics, Microelectronics and Nanotechnology (IEMN), Polytechnic University of Hauts-de-France, University of Lille（电子、微电子与纳米技术研究所（IEMN），法国 Hauts-de-France 工业大学，里尔大学）；Sorbonne Center for Artificial Intelligence, Sorbonne University Abu Dhabi, UAE（索邦人工智能中心，索邦大学阿布扎比校区，阿联酋）
Predicting Brain Responses To Natural Movies With Multimodal LLMs
Medical AI Research Center (MedARC)（医学人工智能研究中心（MedARC））；Psychological and Brain Sciences, Dartmouth College（心理学与脑科学系，达特茅斯学院）；Core for Advanced Magnetic Resonance Imaging (CAMRI), Baylor College of Medicine（先进磁共振成像核心（CAMRI），贝勒医学院）；Sophont；Princeton Neuroscience Institute（普林斯顿神经科学研究所）
