多模态对话中的情感与意图联合理解：一个基准数据集
Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Dataset
AI总结：
本文提出首个多模态对话情感与意图联合理解基准数据集 MC-EIU，涵盖多情感、多意图、多模态和双语言，并设计 EI$^2$ 网络建模情感与意图深层关联，实验验证其有效性。
摘要：
多模态对话中的情感与意图联合理解（MC-EIU）旨在解码多模态对话历史中呈现的语义信息，同时推断当前话语的情感和意图。MC-EIU 是许多人机交互接口的使能技术。然而，在标注、模态、语言多样性和可获取性方面，目前缺乏可用的数据集。在这项工作中，我们提出了一个 MC-EIU 数据集，该数据集具有 7 个情感类别、9 个意图类别、3 种模态（即文本、声学和视觉内容）以及两种语言（即英语和普通话）。此外，它完全开源，可免费获取。据我们所知，MC-EIU 是第一个面向多模态对话的全面且丰富的情感与意图联合理解数据集。随着数据集的发布，我们还开发了一个情感与意图交互（EI$^2$）网络作为参考系统，通过对多模态对话中情感与意图之间的深层相关性进行建模。通过对比实验和消融研究，我们证明了所提出的 EI$^2$ 方法在 MC-EIU 数据集上的有效性。数据集和代码将在以下地址公开：https://github.com/MC-EIU/MC-EIU。
英文摘要：
Emotion and Intent Joint Understanding in Multimodal Conversation (MC-EIU) aims to decode the semantic information manifested in a multimodal conversational history, while inferring the emotions and intents simultaneously for the current utterance. MC-EIU is enabling technology for many human-computer interfaces. However, there is a lack of available datasets in terms of annotation, modality, language diversity, and accessibility. In this work, we propose an MC-EIU dataset, which features 7 emotion categories, 9 intent categories, 3 modalities, i.e., textual, acoustic, and visual content, and two languages, i.e., English and Mandarin. Furthermore, it is completely open-source for free access. To our knowledge, MC-EIU is the first comprehensive and rich emotion and intent joint understanding dataset for multimodal conversation. Together with the release of the dataset, we also develop an Emotion and Intent Interaction (EI$^2$) network as a reference system by modeling the deep correlation between emotion and intent in the multimodal conversation. With comparative experiments and ablation studies, we demonstrate the effectiveness of the proposed EI$^2$ method on the MC-EIU dataset. The dataset and codes will be made available at: https://github.com/MC-EIU/MC-EIU.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
