Multi-task learning is a method for improving the generalizability of multiple tasks. In order to perform multiple classification tasks with one neural network model, the losses of each task should be combined. Previous studies have mostly focused on multiple prediction tasks using joint loss with static weights for training models, choosing the weights between tasks without making sufficient considerations by setting them uniformly or empirically. In this study, we propose a method to calculate joint loss using dynamic weights to improve the total performance, instead of the individual performance, of tasks. We apply this method to design an end-to-end multimodal emotion and gender recognition model using audio and video data. This approach provides proper weights for the loss of each task when the training process ends. In our experiments, emotion and gender recognition with the proposed method yielded a lower joint loss, which is computed as the negative log-likelihood, than using static weights for joint loss. Moreover, our proposed model has better generalizability than other models. To the best of our knowledge, this research is the first to demonstrate the strength of using dynamic weights for joint loss for maximizing overall performance in emotion and gender recognition tasks.
SEGAA: A Unified Approach to Predicting Age, Gender, and Emotion in Speech
School of Computer Science and Engineering(计算机科学与工程学院);Vellore Institute of Technology(韦洛尔理工学院);School of Electronics Engineering(电子工程学院);Voice Intelligence Group(语音智能集团);Samsung R&D Institute(三星研发研究院)
LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences
Beijing University of Posts and Telecommunications(北京邮电大学);East China Normal University(华东师范大学);Shanghai University of International Business and Economics(上海对外经贸大学)
RoHyDR: Robust Hybrid Diffusion Recovery for Incomplete Multimodal Emotion Recognition
School of Computer Science and Engineering(计算机科学与工程学院);South China University of Technology(华南理工大学);School of Future Technology(未来技术学院);University of Oxford(牛津大学);Pengcheng Laboratory(鹏城实验室);Department of Computer Science(计算机科学系)
Touch and Tell: Multimodal Decoding of Human Emotions and Social Gestures for Robots
Faculty of Engineering and Architecture(工程与建筑学院);IDLab-AIRO, Ghent University – imec(IDLab-AIRO,根特大学–imec);Department of Engineering Science, University of Oxford(工程科学系,牛津大学)
EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University(香港理工大学电子与电气工程系);Speech, Language, and Cognition Laboratory, The University of Hong Kong(香港大学语音、语言与认知实验室)