Modality-Collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition
Key Laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, Chinese Academy of Sciences（中国科学院声学研究所语音声学与内容理解重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）
As a vital aspect of affective computing, Multimodal Emotion Recognition has been an active research area in the multimedia community. Despite recent progress, this field still confronts two major challenges in real-world applications: 1) improving the efficiency of constructing joint representations from unaligned multimodal features, and 2) relieving the performance decline caused by random modality feature missing. In this paper, we propose a unified framework, Modality-Collaborative Transformer with Hybrid Feature Reconstruction (MCT-HFR), to address these issues. The crucial component of MCT is a novel attention-based encoder which concurrently extracts and dynamically balances the intra- and inter-modality relations for all associated modalities. With additional modality-wise parameter sharing, a more compact representation can be encoded with less time and space complexity. To improve the robustness of MCT, we further introduce HFR which consists of two modules: Local Feature Imagination (LFI) and Global Feature Alignment (GFA). During model training, LFI leverages complete features as supervisory signals to recover local missing features, while GFA is designed to reduce the global semantic gap between pairwise complete and incomplete representations. Experimental evaluations on two popular benchmark datasets demonstrate that our proposed method consistently outperforms advanced baselines in both complete and incomplete data scenarios.
Multimodal emotion recognition leverages complementary information across modalities to gain performance. However, we cannot guarantee that the data of all modalities are always present in practice. In the studies to predict the missing data across modalities, the inherent difference between heterogeneous modalities, namely the modality gap, presents a challenge. To address this, we propose to use invariant features for a missing modality imagination network (IF-MMIN) which includes two novel mechanisms: 1) an invariant feature learning strategy that is based on the central moment discrepancy …
Modality-Collaborative Low-Rank Decomposers for Few-Shot Video Domain Adaptation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；PengCheng Laboratory（鹏城实验室）
Nanjing University of Science and Technology（南京理工大学）
Human multimodal emotion recognition (MER) aims to perceive human emotions via language, visual and acoustic modalities. Despite the impressive performance of previous MER approaches, the inherent multimodal heterogeneities still haunt and the contribution of different modalities varies significantly. In this work, we mitigate this issue by proposing a decoupled multimodal distillation (DMD) approach that facilitates flexible and adaptive crossmodal knowledge distillation, aiming to enhance the discriminative features of each modality. Specially, the representation of each modality is decouple…
Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective
Jiangsu University（江苏大学）；Griffith University（格里菲斯大学）；National University of Singapore（新加坡国立大学）；University of New South Wales（新南威尔士大学）；Fujian University of Technology（福建理工大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）；Tsinghua University（清华大学）；Technical University of Munich（慕尼黑工业大学）；Imperial College London（伦敦帝国学院）；Queen Mary University of London（伦敦玛丽女王大学）
Sun Yat-sen University（中山大学）；Pazhou Lab（琶洲实验室）；Anhui University（安徽大学）；DarkMatter AI Research（DarkMatter AI 研究院）
Crowd counting is a fundamental yet challenging task, which desires rich information to generate pixel-wise crowd density maps. However, most previous methods only used the limited information of RGB images and cannot well discover potential pedestrians in unconstrained scenarios. In this work, we find that incorporating optical and thermal information can greatly help to recognize pedestrians. To promote future researches in this field, we introduce a large-scale RGBT Crowd Counting (RGBT-CC) benchmark, which contains 2,030 pairs of RGB-thermal images with 138,389 annotated people. Furthermor…
Recently, table structure recognition has achieved impressive progress with the help of deep graph models. Most of them exploit single visual cues of tabular elements or simply combine visual cues with other modalities via early fusion to reason their graph relationships. However, neither early fusion nor individually reasoning in terms of multiple modalities can be appropriate for all varieties of table structures with great diversity. Instead, different modalities are expected to collaborate with each other in different patterns for different table cases. In the community, the importance of …
Collaborative learning is an educational approach that enhances learning through shared goals and working together. Interaction and regulation are two essential factors related to the success of collaborative learning. Since the information from various modalities can reflect the quality of collaboration, a new multimodal dataset with cognitive and emotional triggers is introduced in this paper to explore how regulations affect interactions during the collaborative process. Specifically, a learning task with intentional interventions is designed and assigned to high school students aged 15 yea…
VISTANet: VIsual Spoken Textual Additive Net for Interpretable Multimodal Emotion Recognition
Center for Machine Vision and Signal Analysis, University of Oulu（机器视觉与信号分析中心，奥卢大学）；Indian Institute of Technology Roorkee（印度理工学院罗尔基分校）；State Key Laboratory of Blockchain and Data Security, Zhejiang University（区块链与数据安全国家重点实验室，浙江大学）
Decoupled Hierarchical Distillation for Multimodal Emotion Recognition
School of Computer Science and Engineering, and the Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University（计算机科学与工程学院，新一代人工智能技术及跨学科应用重点实验室，东南大学）；School of Computer Science and Engineering, Nanjing University of Science and Technology（计算机科学与工程学院，南京理工大学）；Institute of Intelligent Information Processing, Taizhou University（智能信息处理研究院，台州大学）；School of Engineering Science, University of Chinese Academy of Sciences（工程科学学院，中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science and Engineering, Nanyang Technological University（计算机科学与工程学院，南洋理工大学）
MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）；Shandong Normal University（山东师范大学）；Renmin University of China（中国人民大学）；Ant Group（蚂蚁集团）；Nanyang Technological University（南洋理工大学）；University of Oulu（奥卢大学）
National University of Singapore（新加坡国立大学）；Singapore University of Technology and Design（新加坡科技设计大学）；Goethe University Frankfurt（法兰克福大学）
Although media content is increasingly produced, distributed, and consumed in multiple combinations of modalities, how individual modalities contribute to the perceived emotion of a media item remains poorly understood. In this paper we present MusicVideos (MuVi), a novel dataset for affective multimedia content analysis to study how the auditory and visual modalities contribute to the perceived emotion of media. The data were collected by presenting music videos to participants in three conditions: music, visual, and audiovisual. Participants annotated the music videos for valence and arousal…
Human beings have rich ways of emotional expressions, including facial action, voice, and natural languages. Due to the diversity and complexity of different individuals, the emotions expressed by various modalities may be semantically irrelevant. Directly fusing information from different modalities may inevitably make the model subject to the noise from semantically irrelevant modalities. To tackle this problem, we propose a multimodal relevance estimation network to capture the relevant semantics among modalities in multimodal emotions. Specifically, we take advantage of an attention mechan…
In this paper, we design a Collaborative-Hierarchical Sparse and Low-Rank (C-HiSLR) model that is natural for recognizing human emotion in visual data. Previous attempts require explicit expression components, which are often unavailable and difficult to recover. Instead, our model exploits the lowrank property over expressive facial frames and rescue inexact sparse representations by incorporating group sparsity. For the CK+ dataset, C-HiSLR on raw expressive faces performs as competitive as the Sparse Representation based Classification (SRC) applied on manually prepared emotions. C-HiSLR pe…
Eurecat, Centre Tecnològic de Catalunya（加泰罗尼亚技术中心Eurecat）；University of Trento（特伦托大学）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）
Emotion recognition is involved in several real-world applications. With an increase in available modalities, automatic understanding of emotions is being performed more accurately. The success in Multimodal Emotion Recognition (MER), primarily relies on the supervised learning paradigm. However, data annotation is expensive, time-consuming, and as emotion expression and perception depends on several factors (e.g., age, gender, culture) obtaining labels with a high reliability is hard. Motivated by these, we focus on unsupervised feature learning for MER. We consider discrete emotions, and as …
DRKF: Decoupled Representations with Knowledge Fusion for Multimodal Emotion Recognition
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；School of Information and Software Engineering, University of Electronic Science and Technology of China（电子科技大学信息与软件工程学院）
University of Chinese Academy of Sciences（中国科学院大学）；Search Lab（360搜索实验室）；Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）
There are threefold challenges in emotion recognition. First, it is difficult to recognize human's emotional states only considering a single modality. Second, it is expensive to manually annotate the emotional data. Third, emotional data often suffers from missing modalities due to unforeseeable sensor malfunction or configuration issues. In this paper, we address all these problems under a novel multi-view deep generative framework. Specifically, we propose to model the statistical relationships of multi-modality emotional data using multiple modality-specific generative networks with a shar…
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学智能自主系统上海研究院）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；School of Mechatronic Engineering and Automation, Shanghai University（上海大学机械电子工程与自动化学院）
CGTGait: Collaborative Graph and Transformer for Gait Emotion Recognition
Huazhong University of Science and Technology, China（华中科技大学）；Wuhan University of Technology, China（武汉理工大学）；Hefei University of Technology, China（合肥工业大学）；Boston University, United States（波士顿大学）
Tampere University（坦佩雷大学）；Aarhus University（奥胡斯大学）
In this paper, we consider the problem of multimodal data analysis with a use case of audiovisual emotion recognition. We propose an architecture capable of learning from raw data and describe three variants of it with distinct modality fusion mechanisms. While most of the previous works consider the ideal scenario of presence of both modalities at all times during inference, we evaluate the robustness of the model in the unconstrained settings where one modality is absent or noisy, and propose a method to mitigate these limitations in a form of modality dropout. Most importantly, we find that…
TACFN: Transformer-based Adaptive Cross-modal Fusion Network for Multimodal Emotion Recognition
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；MTlab, Meitu (China) Limited（美图（中国）有限公司）；Institute of Acoustics, University of Chinese Academy of Sciences（中国科学院声学研究所）
Beijing University of Posts and Telecommunications（北京邮电大学）；East China Normal University（华东师范大学）；Shanghai University of International Business and Economics（上海对外经贸大学）
Learning modality-fused representations and processing unaligned multimodal sequences are meaningful and challenging in multimodal emotion recognition. Existing approaches use directional pairwise attention or a message hub to fuse language, visual, and audio modalities. However, those approaches introduce information redundancy when fusing features and are inefficient without considering the complementarity of modalities. In this paper, we propose an efficient neural network to learn modality-fused representations with CB-Transformer (LMR-CBT) for multimodal emotion recognition from unaligned…
In the latest social networks, more and more people prefer to express their emotions in videos through text, speech, and rich facial expressions. Multimodal video emotion analysis techniques can help understand users' inner world automatically based on human expressions and gestures in images, tones in voices, and recognized natural language. However, in the existing research, the acoustic modality has long been in a marginal position as compared to visual and textual modalities. That is, it tends to be more difficult to improve the contribution of the acoustic modality for the whole multimoda…
Region-aware Spatiotemporal Modeling with Collaborative Domain Generalization for Cross-Subject EEG Emotion Recognition
Laboratory of Digital Image and Intelligent Computation, Shanghai Maritime University, Shanghai 201306, China（上海海洋大学数字图像与智能计算实验室）；Department of Language Science and Technology, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong SAR, China（香港理工大学语言科学与技术系）；School of Chinese as a Second Language, and Center for MRI Research, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, China（北京大学第二语言学院及MRI研究中心）；CAS Key Laboratory of Behavioral Science, Center for Brain Science and Learning Difficulties, Institute of Psychology, Chinese Academy of Sciences, Beijing 100101, China（中国科学院行为科学重点实验室）；Beijing Key Laboratory of Applied Experimental Psychology, Faculty of Psychology, Beijing Normal University, Beijing 100875, China（北京应用实验心理学重点实验室）；Institute of Computing and Intelligence, Harbin Institute of Technology Shenzhen, Shenzhen 518000, China（哈尔滨工业大学深圳校区计算机与智能研究所）
Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition
Great Bay University（大湾大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；City University of Hong Kong (Dongguan)（香港城市大学（东莞））；Tencent Youtu Lab（腾讯优图实验室）
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）
Multi-modal Multi-label Emotion Recognition (MMER) aims to identify various human emotions from heterogeneous visual, audio and text modalities. Previous methods mainly focus on projecting multiple modalities into a common latent space and learning an identical representation for all labels, which neglects the diversity of each modality and fails to capture richer semantic information for each label from different perspectives. Besides, associated relationships of modalities and labels have not been fully exploited. In this paper, we propose versaTile multi-modAl learning for multI-labeL emOti…
