arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2009.07408cs.CL
收藏

面向端任务的结构感知Transformer语言模型的改进

Retrofitting Structure-aware Transformer Language Model for End Tasks

Wuhan University(武汉大学);Guangdong University of Foreign Studies(广东外语外贸大学)

Hao Fei, Yafeng Ren, Donghong Ji

摘要:

我们考虑改造结构感知的transformer为基础的语言模型,以促进最终任务,建议利用句法距离编码到语言模型的短语选区和依存关系连接。采用中间层结构化学习策略进行结构整合,在多任务学习模式下进行主要语义任务训练。实验结果表明,改进后的结构感知Transformer语言模型在提高复杂度的同时,能够准确地归纳出句法短语.通过执行结构感知的微调,我们的模型实现了语义和句法依赖的任务显着改善。

英文摘要:

We consider retrofitting structure-aware Transformer-based language model for facilitating end tasks by proposing to exploit syntactic distance to encode both the phrasal constituency and dependency connection into the language model. A middle-layer structural learning strategy is leveraged for structure integration, accomplished with main semantic task training under multi-task learning scheme. Experimental results show that the retrofitted structure-aware Transformer language model achieves improved perplexity, meanwhile inducing accurate syntactic phrases. By performing structure-aware fine-tuning, our model achieves significant improvements for both semantic- and syntactic-dependent tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑