arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.05729cs.CL
收藏

CPT:一个用于汉语语言理解和生成的预训练非平衡Transformer

CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation

Fudan University(复旦大学);Zhejiang Lab(之江实验室)

Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Hang Yan, Fei Yang, Li Zhe, Hujun Bao, Xipeng Qiu

摘要:

在本文中,我们利用以前的预训练模型(PTM),提出了一种新的中国预训练不平衡Transformer(CPT)。与以往的中文翻译机不同,CPT的设计是利用自然语言理解(NLU)和自然语言生成(NLG)之间的共享知识来提高性能。CPT由三部分组成:共享编码器、理解解码器和生成解码器。具有共享编码器的两个特定解码器分别使用掩码语言建模(MLM)和去噪自动编码(DAE)任务进行预训练。通过部分共享的架构和多任务预训练,CPT可以(1)使用两个解码器学习NLU或NLG任务的特定知识,(2)灵活地进行微调,充分利用模型的潜力。而且不平衡的Transformer节省了计算和存储成本,使CPT具有竞争力,大大加快了文本生成的推理。在汉语自然语言理解和自然语言学习任务上的实验结果表明了CPT的有效性。

英文摘要:

In this paper, we take the advantage of previous pre-trained models (PTMs) and propose a novel Chinese Pre-trained Unbalanced Transformer (CPT). Different from previous Chinese PTMs, CPT is designed to utilize the shared knowledge between natural language understanding (NLU) and natural language generation (NLG) to boost the performance. CPT consists of three parts: a shared encoder, an understanding decoder, and a generation decoder. Two specific decoders with a shared encoder are pre-trained with masked language modeling (MLM) and denoising auto-encoding (DAE) tasks, respectively. With the partially shared architecture and multi-task pre-training, CPT can (1) learn specific knowledge of both NLU or NLG tasks with two decoders and (2) be fine-tuned flexibly that fully exploits the potential of the model. Moreover, the unbalanced Transformer saves the computational and storage cost, which makes CPT competitive and greatly accelerates the inference of text generation. Experimental results on a wide range of Chinese NLU and NLG tasks show the effectiveness of CPT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑