ZRIGF：一种用于零资源图像基础对话生成的创新多模态框架
ZRIGF: An Innovative Multimodal Framework for Zero-Resource Image-Grounded Dialogue Generation
AI总结：
针对零资源场景下图像与文本模态差异导致视觉信息难以利用的问题，提出多模态框架 ZRIGF，通过对比预训练与生成式预训练两阶段策略对齐特征并生成响应，实验证明其在零资源图像基础对话中具有高效性和强泛化能力。
摘要：
图像基础对话系统极大地受益于整合视觉信息，从而实现高质量的响应生成。然而，当前模型在零资源场景中难以有效利用此类信息，主要由于图像与文本模态之间的差异。为了克服这一挑战，我们提出了一种名为 ZRIGF 的创新多模态框架，该框架在零资源情况下吸收图像基础信息以进行对话生成。ZRIGF 实施了两阶段学习策略，包括对比预训练和生成式预训练。对比预训练包含一个文本-图像匹配模块，将图像和文本映射到统一的编码向量空间，以及一个文本辅助的掩蔽图像建模模块，该模块保留预训练视觉特征并促进进一步的多模态特征对齐。生成式预训练采用多模态融合模块和信息转移模块，基于协调的多模态表示产生有见地的响应。在基于文本和图像基础的对话数据集上进行的综合实验证明了 ZRIGF 在生成上下文相关且信息丰富的响应方面的有效性。此外，我们在图像基础对话数据集中采用了完全零资源的场景，以证明我们的框架在新领域中的强大泛化能力。代码可在 https://github.com/zhangbo-nlp/ZRIGF 获取。
英文摘要：
Image-grounded dialogue systems benefit greatly from integrating visual information, resulting in high-quality response generation. However, current models struggle to effectively utilize such information in zero-resource scenarios, mainly due to the disparity between image and text modalities. To overcome this challenge, we propose an innovative multimodal framework, called ZRIGF, which assimilates image-grounded information for dialogue generation in zero-resource situations. ZRIGF implements a two-stage learning strategy, comprising contrastive pre-training and generative pre-training. Contrastive pre-training includes a text-image matching module that maps images and texts into a unified encoded vector space, along with a text-assisted masked image modeling module that preserves pre-training visual features and fosters further multimodal feature alignment. Generative pre-training employs a multimodal fusion module and an information transfer module to produce insightful responses based on harmonized multimodal representations. Comprehensive experiments conducted on both text-based and image-grounded dialogue datasets demonstrate ZRIGF's efficacy in generating contextually pertinent and informative responses. Furthermore, we adopt a fully zero-resource scenario in the image-grounded dialogue dataset to demonstrate our framework's robust generalization capabilities in novel domains. The code is available at https://github.com/zhangbo-nlp/ZRIGF.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
