EmotionHallucer：评估多模态大语言模型中的情绪幻觉
EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Models
AI总结：
本研究推出首个评估MLLMs情绪幻觉的基准EmotionHallucer，从情绪心理学知识和多模态感知双维度、采用对抗式二元QA框架评估38个模型，发现普遍存在情绪幻觉问题，并提出PEP-MEK框架使检测性能平均提升9.90%。
摘要：
情绪理解是一项关键却极具挑战性的任务。多模态大语言模型（MLLMs）的最新进展显著提升了其在该领域的能力，但MLLMs常受幻觉问题困扰，会生成无关或无意义的内容。据我们所知，尽管这一问题至关重要，但目前尚无专门工作评估MLLMs中与情绪相关的幻觉。本研究推出EmotionHallucer，这是首个用于检测与分析MLLMs中情绪幻觉的基准。与人类的情绪理解源于生物机制和社会学习的相互作用不同，MLLMs仅依赖数据驱动的学习，缺乏先天的情绪本能。幸运的是，情绪心理学为人类情绪研究提供了扎实的知识基础。基于此，我们从两个维度评估情绪幻觉：情绪心理学知识与现实世界的多模态感知。为保障评估的稳健性，我们采用对抗式二元问答（QA）框架，通过精心设计的基础问题对与幻觉问题对，评估MLLMs的情绪幻觉倾向。通过在EmotionHallucer上评估38个大语言模型（LLMs）和多模态大语言模型（MLLMs），我们发现：一、当前大多数模型都存在严重的情绪幻觉问题；二、闭源模型在情绪幻觉检测方面的表现优于开源模型，且推理能力能带来额外优势；三、现有模型在情绪心理学知识维度的表现优于多模态情绪感知维度。作为附带成果，这些发现启发我们提出了PEP-MEK框架，在所选模型上，该框架使情绪幻觉检测的平均性能提升了9.90%。相关资源将发布于https://github.com/xxtars/EmotionHallucer。
英文摘要：
Emotion understanding is a critical yet challenging task. Recent advances in Multimodal Large Language Models (MLLMs) have significantly enhanced their capabilities in this area. However, MLLMs often suffer from hallucinations, generating irrelevant or nonsensical content. To the best of our knowledge, despite the importance of this issue, there has been no dedicated effort to evaluate emotion-related hallucinations in MLLMs. In this work, we introduce EmotionHallucer, the first benchmark for detecting and analyzing emotion hallucinations in MLLMs. Unlike humans, whose emotion understanding stems from the interplay of biology and social learning, MLLMs rely solely on data-driven learning and lack innate emotional instincts. Fortunately, emotion psychology provides a solid foundation of knowledge about human emotions. Building on this, we assess emotion hallucinations from two dimensions: emotion psychology knowledge and real-world multimodal perception. To support robust evaluation, we utilize an adversarial binary question-answer (QA) framework, which employs carefully crafted basic and hallucinated pairs to assess the emotion hallucination tendencies of MLLMs. By evaluating 38 LLMs and MLLMs on EmotionHallucer, we reveal that: i) most current models exhibit substantial issues with emotion hallucinations; ii) closed-source models outperform open-source ones in detecting emotion hallucinations, and reasoning capability provides additional advantages; iii) existing models perform better in emotion psychology knowledge than in multimodal emotion perception. As a byproduct, these findings inspire us to propose the PEP-MEK framework, which yields an average improvement of 9.90% in emotion hallucination detection across selected models. Resources will be available at https://github.com/xxtars/EmotionHallucer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
