Recent works on multi-modal emotion recognition move towards end-to-end models, which can extract the task-specific features supervised by the target task compared with the two-phase pipeline. However, previous methods only model the feature interactions between the textual and either acoustic and visual modalities, ignoring capturing the feature interactions between the acoustic and visual modalities. In this paper, we propose the multi-modal end-to-end transformer (ME2ET), which can effectively model the tri-modal features interaction among the textual, acoustic, and visual modalities at the low-level and high-level. At the low-level, we propose the progressive tri-modal attention, which can model the tri-modal feature interactions by adopting a two-pass strategy and can further leverage such interactions to significantly reduce the computation and memory complexity through reducing the input token length. At the high-level, we introduce the tri-modal feature fusion layer to explicitly aggregate the semantic representations of three modalities. The experimental results on the CMU-MOSEI and IEMOCAP datasets show that ME2ET achieves the state-of-the-art performance. The further in-depth analysis demonstrates the effectiveness, efficiency, and interpretability of the proposed progressive tri-modal attention, which can help our model to achieve better performance while significantly reducing the computation and memory cost. Our code will be publicly available.
HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech
South China University of Technology, China（华南理工大学，中国）；Huya Inc., China（欢丫公司，中国）；Tongyi Fun Team, Alibaba Group, China（通义 Fun 团队，阿里巴巴集团，中国）；Foshan University, China（佛山大学，中国）
Span-level Emotion-Cause-Category Triplet Extraction with Instruction Tuning LLMs and Data Augmentation
College of Computer Science and Engineering（计算机科学与工程学院）；Shandong University of Science and Technology（山东科技大学）；Guangxi Key Lab of Human-machine Interaction and Intelligent Decision（广西人机交互与智能决策重点实验室）；Nanning Normal University（南宁师范大学）
Contrast and Generation Make BART a Good Dialogue Emotion Recognizer
School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Peng Cheng Laboratory（鹏城实验室）；Shanghai Key Laboratory of Intelligent Information Processing, Fudan University（复旦大学上海市智能信息处理重点实验室）
Empathetic Dialogue Generation via Sensitive Emotion Recognition and Sensible Knowledge Selection
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；WeChat AI, Tencent Inc（腾讯公司微信AI）
