多模态对话中的生成式情感原因解释
Generative Emotion Cause Explanation in Multimodal Conversations
AI总结：
本文提出多模态对话情感原因解释任务MECEC，构建ECEM数据集，并设计FAME-Net利用大语言模型分析面部表情和情感传染效应，在实验中优于多个基线模型。
摘要：
多模态对话作为人类交流的重要形式，承载着丰富的情感内容，因此探索其中情感产生的原因是一项具有重要研究意义的工作。然而，现有关于情感原因的研究通常采用单一文本模态中的话语选择方法来定位原因话语。这种方法仍局限于粗粒度评估，缺乏对情感成因的细致解释，并且在识别多模态情感触发因素方面能力不足。为此，我们引入了一项任务——多模态对话情感原因解释（MECEC）。该任务旨在基于对话的多模态上下文生成摘要，清晰直观地描述触发特定情感的原因。为适应该任务，我们在MELD数据集的基础上构建了一个新数据集（ECEM）。ECEM将视频片段与人物情感的详细解释相结合，有助于探索多模态对话中情感表达背后的因果因素。我们进一步提出了一种新方法FAME-Net，该方法利用大语言模型（LLMs）的能力分析视觉数据，并准确解读视频中通过面部表情传达的情感。通过利用面部情感的传染效应，FAME-Net能够有效捕捉对话参与者的情感原因。我们在新构建数据集上的实验结果表明，FAME-Net优于多个优秀的基线模型。代码和数据集可在https://github.com/3222345200/FAME-Net获取。
英文摘要：
Multimodal conversation, a crucial form of human communication, carries rich emotional content, making the exploration of the causes of emotions within it a research endeavor of significant importance. However, existing research on the causes of emotions typically employs an utterance selection method within a single textual modality to locate causal utterances. This approach remains limited to coarse-grained assessments, lacks nuanced explanations of emotional causation, and demonstrates inadequate capability in identifying multimodal emotional triggers. Therefore, we introduce a task-\textbf{Multimodal Emotion Cause Explanation in Conversation (MECEC)}. This task aims to generate a summary based on the multimodal context of conversations, clearly and intuitively describing the reasons that trigger a given emotion. To adapt to this task, we develop a new dataset (ECEM) based on the MELD dataset. ECEM combines video clips with detailed explanations of character emotions, helping to explore the causal factors behind emotional expression in multimodal conversations. A novel approach, FAME-Net, is further proposed, that harnesses the power of Large Language Models (LLMs) to analyze visual data and accurately interpret the emotions conveyed through facial expressions in videos. By exploiting the contagion effect of facial emotions, FAME-Net effectively captures the emotional causes of individuals engaged in conversations. Our experimental results on the newly constructed dataset show that FAME-Net outperforms several excellent baselines. Code and dataset are available at https://github.com/3222345200/FAME-Net.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
