In this study, we investigate multimodal foundation models (MFMs) for emotion recognition from non-verbal sounds. We hypothesize that MFMs, with their joint pre-training across multiple modalities, will be more effective in non-verbal sounds emotion recognition (NVER) by better interpreting and differentiating subtle emotional cues that may be ambiguous in audio-only foundation models (AFMs). To validate our hypothesis, we extract representations from state-of-the-art (SOTA) MFMs and AFMs and evaluated them on benchmark NVER datasets. We also investigate the potential of combining selected foundation model representations to enhance NVER further inspired by research in speech recognition and audio deepfake detection. To achieve this, we propose a framework called MATA (Intra-Modality Alignment through Transport Attention). Through MATA coupled with the combination of MFMs: LanguageBind and ImageBind, we report the topmost performance with accuracies of 76.47%, 77.40%, 75.12% and F1-scores of 70.35%, 76.19%, 74.63% for ASVP-ESD, JNV, and VIVAE datasets against individual FMs and baseline fusion techniques and report SOTA on the benchmark datasets.
PARROT: Synergizing Mamba and Attention-based SSL Pre-Trained Models via Parallel Branch Hadamard Optimal Transport for Speech Emotion Recognition
IIIT-DelhiIndia(印度IIIT-德尔hi大学);V.B.S.P.UIndia(印度V.B.S.P.U大学);UPESIndia(印度UPES大学);Independent ResearcherIndia(印度独立研究者);University of TartuEstonia(爱沙尼亚塔尔图大学);Plaksha UniversityIndia(印度Plaksha大学)
CARE: A Multimodal Corpus for Studying Speech and Non-Verbal Communication Across Multiple Medical Conditions
PRHLT, Universitat Politècnica de València(巴伦西亚理工大学);Sword Health(剑健康);INESC-ID(葡萄牙信息与通信系统研究所);Instituto Superior Técnico, Universidade de Lisboa(里斯本大学技术学院)
Make Acoustic and Visual Cues Matter: CH-SIMS v2.0 Dataset and AV-Mixup Consistent Module
Tsinghua University(清华大学);Hebei University of Science and Technology(河北科技大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Beijing University of Posts and Telecommunications(北京邮电大学);China Agricultural University(中国农业大学)
ParaLBench: A Large-Scale Benchmark for Computational Paralinguistics over Acoustic Foundation Models
College of Computer Science and Electronic Engineering, Hunan University(湖南大学计算机与电子工程学院);Shenzhen Research Institute, Hunan University(湖南大学深圳研究院);School of Information Science and Engineering, Hunan Normal University(湖南师范大学信息科学与工程学院);UBTECH Robotics Corp(优必选机器人公司)