准确理解情感对于人机交互等领域至关重要。由于情感的复杂性及其多模态本质（例如，情感会受到面部表情和音频的影响），研究人员已转向使用多模态模型而非单一模态来理解人类情感。然而，当前的视频多模态大语言模型（MLLM）在有效整合音频以及识别细微面部微表情方面遇到困难。此外，缺乏详细的情感分析数据集也限制了多模态情感分析的发展。为解决这些问题，我们引入了一个自审数据集和一个人工审校数据集，分别包含24,137个粗粒度样本和3,500个带有详细情感标注的人工标注样本。这些数据集使模型能够从多样化场景中学习，并更好地泛化到现实世界应用。此外，除音频建模外，我们提出将面部编码模型显式整合到现有的先进 Video MLLM 中，使该 MLLM 能够有效统一音频与细微面部线索以进行情感理解。通过在统一空间中对齐这些特征，并在我们提出的数据集上采用指令微调，我们的 Omni-Emotion 在情感识别和推理任务中均达到最先进性能。
英文摘要：
Understanding emotions accurately is essential for fields like human-computer interaction. Due to the complexity of emotions and their multi-modal nature (e.g., emotions are influenced by facial expressions and audio), researchers have turned to using multi-modal models to understand human emotions rather than single-modality. However, current video multi-modal large language models (MLLMs) encounter difficulties in effectively integrating audio and identifying subtle facial micro-expressions. Furthermore, the lack of detailed emotion analysis datasets also limits the development of multimodal emotion analysis. To address these issues, we introduce a self-reviewed dataset and a human-reviewed dataset, comprising 24,137 coarse-grained samples and 3,500 manually annotated samples with detailed emotion annotations, respectively. These datasets allow models to learn from diverse scenarios and better generalize to real-world applications. Moreover, in addition to the audio modeling, we propose to explicitly integrate facial encoding models into the existing advanced Video MLLM, enabling the MLLM to effectively unify audio and the subtle facial cues for emotion understanding. By aligning these features within a unified space and employing instruction tuning in our proposed datasets, our Omni-Emotion achieves state-of-the-art performance in both emotion recognition and reasoning tasks.
OmniSpace: Efficient Geometry Awareness for Autonomous Vehicles MLLMs
University of Arkansas（阿肯色大学）；Google Research, Google（谷歌研究院）；University of Liverpool（利物浦大学）；Max Planck Research School for Intelligent Systems（马克斯·普朗克智能系统研究所）
Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；VCIP & TMCC & DISSec, College of Computer Science, Nankai University（南开大学计算机学院）；Department of Psychological and Cognitive Sciences, Tsinghua University（清华大学心理学与认知科学系）；University of Chinese Academy of Sciences（中国科学院大学）
EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Models
Lappeenranta-Lahti University of Technology LUT（拉佩兰塔-拉赫蒂技术大学）；University of Oulu（奥卢大学）；Southeast University（东南大学）；Shanghai Jiao Tong University（上海交通大学）；Brno University of Technology（布拉格技术大学）
OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks
Chinese University of Hong Kong（香港中文大学）；Sun Yat-sen Memorial Hospital, Sun Yat-sen University（中山大学孙逸仙纪念医院）；Southern Medical University（南方医科大学）；Zhongshan Hospital, Fudan University（复旦大学中山医院）；Department of Neurosurgery, Prince of Wales Hospital（威尔士王室医院神经外科部）
MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models
The Chinese University of Hong Kong（香港中文大学）；Tongyi Lab（通义实验室）；SZTU（深圳技术大学）；Peking University（北京大学）；Tongji University（同济大学）；CASIA（中国科学院自动化所）；Tencent（腾讯）；UCLA（加州大学洛杉矶分校）；Westlake University（西湖大学）；NTU（国立大学）
GRACE: Boosting Video MLLMs with Grounded Action-Centric Evidence for Viewer Sentiment Prediction
Shanghai Jiao Tong University（上海交通大学）；Hangzhou Dianzi University（杭州电子科技大学）；The 52nd Research Institute of China Electronics Technology Group Corporation（中国电子科技集团公司第五十二研究所）；Hangzhou Bywin Technology Co., Ltd.（杭州百威科技有限公司）；Zhejiang Dahua Technology Co., Ltd.（浙江大华技术股份有限公司）；School of Information Science and Engineering, Shandong University（山东大学信息科学与工程学院）；Haihe Laboratory of Information Technology Application Innovation（海河信息技术应用创新实验室）
Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark
Hefei University of Technology（合肥工业大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence (IAI), Hefei Comprehensive National Science Center（人工智能研究院（IAI），合肥综合性国家科学中心）
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Hong Kong University of Science and Technology（香港科技大学）；The University of Hong Kong（香港大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；The Chinese University of Hong Kong（香港中文大学）；Sun Yat-sen University（中山大学）；Southern University of Science and Technology（南方科技大学）
Multimodal Video Emotion Recognition with Reliable Reasoning Priors
Lenovo Research（联想研究院）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；Institute of Automation, CAS（中国科学院自动化研究所）；Macau University of Science and Technology（澳门科学理工学院）；School of Computer Science and Technology, UCAS（中国科学院大学计算机科学与技术学院）
OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shanghai AI Lab（上海人工智能实验室）；Beihang University（北京航空航天大学）；Nanyang Technological University（南洋理工大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Chinese University of Hong Kong（香港中文大学）
