arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.06664cs.CL
收藏

SPACE-3:面向任务的对话理解和生成的统一对话模型预训练

SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generation

Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Alibaba Group(阿里巴巴集团)

Wanwei He, Yinpei Dai, Min Yang, Jian Sun, Fei Huang, Luo Si, Yongbin Li

摘要:

最近,预训练方法在面向任务的对话(TOD)系统中取得了显着的成功。然而,大多数现有的TOD预训练模型专注于对话理解或对话生成,但不是两者兼而有之。在本文中,我们提出了SPACE-3,这是一种新的统一的半监督预训练会话模型,它从具有有限注释的大规模对话语料库中学习,可以在广泛的下游对话任务中进行有效的微调。具体而言,SPACE-3由单个Transformer中的四个连续组件组成,以维持TOD系统中的任务流:(i)对话编码模块,用于对对话历史进行编码,(ii)对话理解模块,用于从用户查询或系统响应中提取语义向量,(iii)对话策略模块,用于生成包含响应的高级语义的策略向量,以及(iv)用于产生适当响应的对话生成模块。我们为每个组件设计了专门的预培训目标。具体地说,我们用跨度掩码语言建模来预训练对话编码模块,以学习上下文化的对话信息。为了捕获结构化对话语义,我们通过一种新的树诱导的半监督对比学习目标与额外的对话注释的帮助下,预训练对话理解模块。此外,我们通过最小化其输出策略向量与响应的语义向量之间的L2距离来预训练对话策略模块,以进行策略优化。最后,通过语言建模对对话生成模型进行预训练。结果表明,SPACE-3在八个下游对话基准测试中达到了最先进的性能,包括意图预测、对话状态跟踪和端到端对话建模。我们还表明,SPACE-3具有更强的Few-Shot能力比现有的模型在低资源设置。

英文摘要:

Recently, pre-training methods have shown remarkable success in task-oriented dialog (TOD) systems. However, most existing pre-trained models for TOD focus on either dialog understanding or dialog generation, but not both. In this paper, we propose SPACE-3, a novel unified semi-supervised pre-trained conversation model learning from large-scale dialog corpora with limited annotations, which can be effectively fine-tuned on a wide range of downstream dialog tasks. Specifically, SPACE-3 consists of four successive components in a single transformer to maintain a task-flow in TOD systems: (i) a dialog encoding module to encode dialog history, (ii) a dialog understanding module to extract semantic vectors from either user queries or system responses, (iii) a dialog policy module to generate a policy vector that contains high-level semantics of the response, and (iv) a dialog generation module to produce appropriate responses. We design a dedicated pre-training objective for each component. Concretely, we pre-train the dialog encoding module with span mask language modeling to learn contextualized dialog information. To capture the structured dialog semantics, we pre-train the dialog understanding module via a novel tree-induced semi-supervised contrastive learning objective with the help of extra dialog annotations. In addition, we pre-train the dialog policy module by minimizing the L2 distance between its output policy vector and the semantic vector of the response for policy optimization. Finally, the dialog generation model is pre-trained by language modeling. Results show that SPACE-3 achieves state-of-the-art performance on eight downstream dialog benchmarks, including intent prediction, dialog state tracking, and end-to-end dialog modeling. We also show that SPACE-3 has a stronger few-shot ability than existing models under the low-resource setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑