BI-MDRG：在多模态对话响应生成中桥接图像历史
BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generation
AI总结：
针对多模态对话响应生成中以文本为中间步骤易丢失图像信息的问题，提出BI-MDRG桥接图像历史信息，提升文本响应相关性与图像物体一致性，还构建了300段标注对话的图像一致性评估数据集。
摘要：
多模态对话响应生成（Multimodal Dialogue Response Generation, MDRG）是近期提出的一项任务，模型需要根据对话上下文生成文本、图像或二者混合形式的响应。由于缺乏专门针对该任务的大规模数据集，且借助强大预训练模型存在诸多优势，现有研究将文本模态作为模型图像输入与输出的中间步骤，而非采用端到端方式。但这种方法会忽略图像的关键信息，阻碍两方面表现：一是基于图像的文本响应质量，二是图像响应中物体的一致性。本文提出BI-MDRG，通过桥接响应生成路径，利用图像历史信息提升文本响应与图像内容的相关性，以及连续图像响应中物体的一致性。在多模态对话基准数据集上开展的大量实验表明，BI-MDRG可有效提升多模态对话的质量。此外，注意到当前评估多模态对话中图像一致性的基准数据集存在缺口，我们构建了一个包含300段经过标注对话的精选数据集，用于追踪对话过程中的物体一致性。
英文摘要：
Multimodal Dialogue Response Generation (MDRG) is a recently proposed task where the model needs to generate responses in texts, images, or a blend of both based on the dialogue context. Due to the lack of a large-scale dataset specifically for this task and the benefits of leveraging powerful pre-trained models, previous work relies on the text modality as an intermediary step for both the image input and output of the model rather than adopting an end-to-end approach. However, this approach can overlook crucial information about the image, hindering 1) image-grounded text response and 2) consistency of objects in the image response. In this paper, we propose BI-MDRG that bridges the response generation path such that the image history information is utilized for enhanced relevance of text responses to the image content and the consistency of objects in sequential image responses. Through extensive experiments on the multimodal dialogue benchmark dataset, we show that BI-MDRG can effectively increase the quality of multimodal dialogue. Additionally, recognizing the gap in benchmark datasets for evaluating the image consistency in multimodal dialogue, we have created a curated set of 300 dialogues annotated to track object consistency across conversations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
