arXiv 2512.193792026-02-10cs.LGcs.AIcs.MM

OmniMER: 增辅增强的LLM适应用于印度尼西亚多模态情感识别

OmniMER: Auxiliary-Enhanced LLM Adaptation for Indonesian Multimodal Emotion Recognition

School of Information Science and Technology, Guangdong University of Foreign Studies（广东外语外贸大学信息科学与技术学院）；School of Computer Science, Guangdong University of Technology（广东工业大学计算机学院）；Faculty of Asian Languages and Cultures, Guangdong University of Foreign Studies（广东外语外贸大学亚洲语言文化学院）；School of Engineering, Westlake University（西湖大学工程学院）；School of Computer Science and Intelligence Education, Lingnan Normal University（岭南师范学院计算机科学与智能教育学院）；School of Automation, Chongqing University（重庆大学自动化学院）

Xueming Yan, Boyan Xu, Yaochu Jin, Lixian Xiao, Wenlong Ye, Runyang Cai, Zeqi Zheng, Jingfa Liu, Aimin Yang, Yongduan Song

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

OmniMER通过三种辅助任务提升印度尼西亚多模态情感识别性能，实现情感分类和识别的显著提升。

摘要：

印度尼西亚语，由超过2亿人使用，尽管在东南亚社交媒体平台上占据主导地位，但在多模态情感识别研究中仍被忽视。我们介绍了IndoMER，首个针对印度尼西亚语的多模态情感识别基准，包含来自203名说话者的1944个视频片段，具有时间对齐的文本、音频和视觉注释，涵盖七种情感类别。该数据集表现出现实挑战，包括跨模态不一致性和由印度尼西亚文化沟通规范塑造的长尾类分布。为解决这些挑战，我们提出了OmniMER，一个基于Qwen2.5-Omni的多模态适应框架，通过三个辅助模态特定感知任务增强情感识别：文本中的情感关键词提取、视频中的面部表情分析和音频中的语调分析。这些辅助任务帮助模型在融合前识别每种模态中的情感相关线索，减少低资源设置中对虚假相关性的依赖。在IndoMER上的实验表明，OmniMER在情感分类上达到0.582的Macro-F1，在情感识别上达到0.454，分别比基模型高出7.6和22.1个绝对点。跨语言评估在中文CH-SIMS数据集上进一步展示了所提出框架的通用性。数据集和代码已公开可用。https://github.com/yanxm01/INDOMER

英文摘要：

Indonesian, spoken by over 200 million people, remains underserved in multimodal emotion recognition research despite its dominant presence on Southeast Asian social media platforms. We introduce IndoMER, the first multimodal emotion recognition benchmark for Indonesian, comprising 1,944 video segments from 203 speakers with temporally aligned text, audio, and visual annotations across seven emotion categories. The dataset exhibits realistic challenges including cross-modal inconsistency and long-tailed class distributions shaped by Indonesian cultural communication norms. To address these challenges, we propose OmniMER, a multimodal adaptation framework built upon Qwen2.5-Omni that enhances emotion recognition through three auxiliary modality-specific perception tasks: emotion keyword extraction for text, facial expression analysis for video, and prosody analysis for audio. These auxiliary tasks help the model identify emotion-relevant cues in each modality before fusion, reducing reliance on spurious correlations in low-resource settings. Experiments on IndoMER show that OmniMER achieves 0.582 Macro-F1 on sentiment classification and 0.454 on emotion recognition, outperforming the base model by 7.6 and 22.1 absolute points respectively. Cross-lingual evaluation on the Chinese CH-SIMS dataset further demonstrates the generalizability of the proposed framework. The dataset and code are publicly available. https://github.com/yanxm01/INDOMER

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。