arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.04692cs.CL
收藏

不是所有的注意力都是你需要的

Not All Attention Is All You Need

Shanghai Jiao Tong University(上海交通大学);Soochow University(苏州大学)

Hongqiu Wu, Hai Zhao, Min Zhang

摘要:

除了在最近的自然语言处理中预先训练的语言模型(PrLM)的成功故事之外,由于不寻常的大模型大小,它们容易过度拟合。为此,辍学作为一种治疗。然而,现有的方法,如基于随机的,基于知识的和基于搜索的丢弃更普遍,但对基于自我注意力的模型,这是广泛选择的基本架构的PrLM的有效性较低。在本文中,我们提出了一种新的dropout方法AttendOut,让自我注意力授权的PrLM能够更强大的特定于任务的调整。我们证明了具有精心训练设计的最先进模型可以实现更强大的结果。我们验证了广泛的自然语言处理任务,我们的方法的普遍性。

英文摘要:

Beyond the success story of pre-trained language models (PrLMs) in recent natural language processing, they are susceptible to over-fitting due to unusual large model size. To this end, dropout serves as a therapy. However, existing methods like random-based, knowledge-based and search-based dropout are more general but less effective onto self-attention based models, which are broadly chosen as the fundamental architecture of PrLMs. In this paper, we propose a novel dropout method named AttendOut to let self-attention empowered PrLMs capable of more robust task-specific tuning. We demonstrate that state-of-the-art models with elaborate training design may achieve much stronger results. We verify the universality of our approach on extensive natural language processing tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑