arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.09146cs.CL
收藏

基于语义的对话理解预训练

Semantic-based Pre-training for Dialogue Understanding

Tencent AI Lab(腾讯人工智能实验室);Westlake University(西湖大学);Westlake Institute for Advanced Study(西湖高等研究院)

Xuefeng Bai, Linfeng Song, Yue Zhang

摘要:

预训练的语言模型在对话任务上取得了很大的进展。然而,这些模型通常是在表面对话文本上训练的,因此被证明在理解对话上下文的主要语义方面是薄弱的。我们研究了抽象意义表征(AMR)作为预训练模型的显式语义知识,以在预训练期间捕获对话中的核心语义信息。特别地,我们提出了一种基于语义的预训练框架,该框架扩展了标准预训练框架(Devlin等人,2019)通过三个任务学习1)核心语义单元,2)语义关系和3)根据AMR图的整体语义表示。对聊天和任务型对话的理解实验表明了该模型的优越性。据我们所知,我们是第一个利用深度语义表示进行对话预训练的公司。

英文摘要:

Pre-trained language models have made great progress on dialogue tasks. However, these models are typically trained on surface dialogue text, thus are proven to be weak in understanding the main semantic meaning of a dialogue context. We investigate Abstract Meaning Representation (AMR) as explicit semantic knowledge for pre-training models to capture the core semantic information in dialogues during pre-training. In particular, we propose a semantic-based pre-training framework that extends the standard pre-training framework (Devlin et al., 2019) by three tasks for learning 1) core semantic units, 2) semantic relations and 3) the overall semantic representation according to AMR graphs. Experiments on the understanding of both chit-chats and task-oriented dialogues show the superiority of our model. To our knowledge, we are the first to leverage a deep semantic representation for dialogue pre-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑