IMAD：图像增强的多模态对话
IMAD: IMage-Augmented multi-modal Dialogue
AI总结：
针对对话系统难以利用视觉信息且缺乏相关数据集的问题，本文提出两阶段自动构建方法，创建 IMAD 多模态对话数据集，并训练出优于无图像模型和 BlenderBot 的基线。
摘要：
目前，对话系统在处理基于文本的交流方面已经取得了高性能。然而，它们尚未有效地整合视觉信息，这构成了一项重大挑战。此外，现有在对话生成中纳入图像的模型主要关注讨论图像本身。我们提出的方法为多模态对话系统提供了一种新视角，即在对话语境中解释图像。通过这样做，我们旨在扩展当前对话系统的能力，并推动其从单一模态（文本）转向多模态。然而，该任务缺乏经过验证的、同时包含图像和对话语境的英语数据集。因此，我们提出了一种两阶段方法来自动构建多模态对话数据集。第一阶段，我们利用文本到图像相似度和句子相似度来识别哪些话语可以被图像替换。第二阶段，我们通过选择相关图像子集并使用视觉问答模型进行过滤，来替换这些话语。我们使用该方法并结合额外标注，创建了 IMage Augmented multi-modal Dialogue 数据集（IMAD），可作为该任务的经验证数据集。此外，我们提出了一个在该数据集上训练的基线模型，其表现优于在相同无图像数据上训练的模型以及 BlenderBot。
英文摘要：
Currently, dialogue systems have achieved high performance in processing text-based communication. However, they have not yet effectively incorporated visual information, which poses a significant challenge. Furthermore, existing models that incorporate images in dialogue generation focus on discussing the image itself. Our proposed approach presents a novel perspective on multi-modal dialogue systems, which interprets the image in the context of the dialogue. By doing so, we aim to expand the capabilities of current dialogue systems and transition them from single modality (text) to multi-modality. However, there is a lack of validated English datasets that contain both images and dialogue contexts for this task. Thus, we propose a two-stage approach to automatically construct a multi-modal dialogue dataset. In the first stage, we utilize text-to-image similarity and sentence similarity to identify which utterances could be replaced with an image. In the second stage, we replace those utterances by selecting a subset of relevant images and filtering them with a visual question answering model. We used this approach, along with additional labeling, to create the IMage Augmented multi-modal Dialogue dataset (IMAD), which can serve as a validated dataset for this task. Furthermore, we propose a baseline model trained on this dataset, which outperforms model trained on the same data without images and BlenderBot.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
