arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2207.07934cs.CLcs.HCcs.MM
收藏

采用双重知识增强生成式预训练语言模型的多模态对话系统

Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model

School of Computer Science and Technology, Shandong University(山东大学计算机科学与技术学院);School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院)

Xiaolin Chen, Xuemeng Song, Liqiang Jing, Shuo Li, Linmei Hu, Liqiang Nie

AI总结:

针对多模态任务型对话中文本回复生成忽视生成式预训练与上下文知识的问题,提出DKMD模型,通过双重知识选择、知识增强上下文学习及改进BART解码器融合文本与视觉知识,实验证明其优于现有方法。

摘要:

面向多模态任务型对话系统的文本回复生成,旨在给定多模态上下文的情况下生成恰当的文本回复,是一项必不可少但颇具挑战性的任务。尽管现有努力已经取得了令人瞩目的成功,但它们仍存在两个关键局限:1)忽视了生成式预训练的益处;2)忽略了与文本上下文相关的知识。为解决这些局限,我们提出了一种新颖的用于多模态任务型对话系统的双重知识增强生成式预训练语言模型(DKMD),该模型由三个关键组件构成:双重知识选择、双重知识增强上下文学习以及知识增强回复生成。具体而言,双重知识选择组件旨在根据给定上下文的文本和视觉两种模态选择相关知识。随后,双重知识增强上下文学习组件的目标是从全局和局部两个视角将所选知识无缝整合到多模态上下文学习中,同时还会挖掘跨模态语义关系。此外,知识增强回复生成组件包含一个经过改进的 BART 解码器,其中引入了一个额外的点积知识-解码器注意力子层,用于显式利用知识以推进文本回复生成。在一个公开数据集上进行的大量实验验证了所提出的 DKMD 相对于最先进竞争方法的优越性。

英文摘要:

Text response generation for multimodal task-oriented dialog systems, which aims to generate the proper text response given the multimodal context, is an essential yet challenging task. Although existing efforts have achieved compelling success, they still suffer from two pivotal limitations: 1) overlook the benefit of generative pre-training, and 2) ignore the textual context related knowledge. To address these limitations, we propose a novel dual knowledge-enhanced generative pretrained language model for multimodal task-oriented dialog systems (DKMD), consisting of three key components: dual knowledge selection, dual knowledge-enhanced context learning, and knowledge-enhanced response generation. To be specific, the dual knowledge selection component aims to select the related knowledge according to both textual and visual modalities of the given context. Thereafter, the dual knowledge-enhanced context learning component targets seamlessly integrating the selected knowledge into the multimodal context learning from both global and local perspectives, where the cross-modal semantic relation is also explored. Moreover, the knowledge-enhanced response generation component comprises a revised BART decoder, where an additional dot-product knowledge-decoder attention sub-layer is introduced for explicitly utilizing the knowledge to advance the text response generation. Extensive experiments on a public dataset verify the superiority of the proposed DKMD over state-of-the-art competitors.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑