arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.08650cs.CL
收藏

照片分享多模态对话生成的端到端模型

An End-to-End Model for Photo-Sharing Multi-modal Dialogue Generation

Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));The University of Edinburgh(爱丁堡大学)

Peiming Guo, Sinuo Liu, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, Min Zhang

AI总结:

本文提出首个照片分享多模态对话生成的端到端模型,集成图像感知与生成于大语言模型,通过动态词汇变换矩阵及直通和Gumbel-Softmax技术实现端到端梯度传播,在PhotoChat和DialogCC数据集上取得最先进性能。

摘要:

照片分享多模态对话生成要求对话代理不仅生成文本回复,还要在适当的时候分享照片。以图像文本描述作为桥梁,一种流水线模型集成了图像描述模型、文本生成模型和图像生成模型来处理这一复杂的多模态任务。然而,用文本描述表示图像可能会丢失重要的视觉细节和信息,并在复杂的对话系统中导致错误传播。此外,流水线模型将三个模型分离,因为离散的图像文本描述阻碍了端到端的梯度传播。我们提出了第一个用于照片分享多模态对话生成的端到端模型,该模型将图像感知器和图像生成器与大型语言模型集成在一起。大型语言模型在输入端使用Q-Former来感知视觉图像。对于输出端的图像生成,我们提出了一种动态词汇变换矩阵,并使用直通(straight-through)和Gumbel-Softmax技术来对齐大型语言模型和稳定扩散模型,实现端到端的梯度传播。我们在PhotoChat和DialogCC数据集上进行了实验来评估我们的端到端模型。与流水线模型相比,端到端模型在文本和图像生成的各项指标上取得了最先进的性能。更多的分析实验也验证了端到端模型在照片分享多模态对话生成中的有效性。

英文摘要:

Photo-Sharing Multi-modal dialogue generation requires a dialogue agent not only to generate text responses but also to share photos at the proper moment. Using image text caption as the bridge, a pipeline model integrates an image caption model, a text generation model, and an image generation model to handle this complex multi-modal task. However, representing the images with text captions may loss important visual details and information and cause error propagation in the complex dialogue system. Besides, the pipeline model isolates the three models separately because discrete image text captions hinder end-to-end gradient propagation. We propose the first end-to-end model for photo-sharing multi-modal dialogue generation, which integrates an image perceptron and an image generator with a large language model. The large language model employs the Q-Former to perceive visual images in the input end. For image generation in the output end, we propose a dynamic vocabulary transformation matrix and use straight-through and gumbel-softmax techniques to align the large language model and stable diffusion model and achieve end-to-end gradient propagation. We perform experiments on PhotoChat and DialogCC datasets to evaluate our end-to-end model. Compared with pipeline models, the end-to-end model gains state-of-the-art performances on various metrics of text and image generation. More analysis experiments also verify the effectiveness of the end-to-end model for photo-sharing multi-modal dialogue generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑