Photo-Sharing Multi-modal dialogue generation requires a dialogue agent not only to generate text responses but also to share photos at the proper moment. Using image text caption as the bridge, a pipeline model integrates an image caption model, a text generation model, and an image generation model to handle this complex multi-modal task. However, representing the images with text captions may loss important visual details and information and cause error propagation in the complex dialogue system. Besides, the pipeline model isolates the three models separately because discrete image text captions hinder end-to-end gradient propagation. We propose the first end-to-end model for photo-sharing multi-modal dialogue generation, which integrates an image perceptron and an image generator with a large language model. The large language model employs the Q-Former to perceive visual images in the input end. For image generation in the output end, we propose a dynamic vocabulary transformation matrix and use straight-through and gumbel-softmax techniques to align the large language model and stable diffusion model and achieve end-to-end gradient propagation. We perform experiments on PhotoChat and DialogCC datasets to evaluate our end-to-end model. Compared with pipeline models, the end-to-end model gains state-of-the-art performances on various metrics of text and image generation. More analysis experiments also verify the effectiveness of the end-to-end model for photo-sharing multi-modal dialogue generation.
Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding
Vector Institute for Artificial Intelligence(矢量人工智能研究所);University of Toronto(多伦多大学);Peter Munk Cardiac Centre, University Health Network(大学健康网络彼得·蒙克心脏中心);McGill University(麦吉尔大学);University Health Network(大学健康网络)
NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews
University of California, Berkeley(加州大学伯克利分校);University of Southern California(南加州大学);Information Sciences Institute(信息科学研究所);Northeastern University(东北大学)
Context-Driven Incremental Compression for Multi-Turn Dialogue Generation
The Hong Kong University of Science and Technology(香港科技大学);NVIDIA AI Technology Center(NVIDIA AI技术中心);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学)
Adaptive Parameterization for Neural Dialogue Generation
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Data Science Lab, JD.com(京东数据科学实验室)