arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1912.00544cs.CLcs.LG
收藏

多尺度自注意文本分类

Multi-Scale Self-Attention for Text Classification

Fudan University(复旦大学);AWS Shanghai AI Lab(AWS上海人工智能实验室);New York University Shanghai(上海纽约大学)

Qipeng Guo, Xipeng Qiu, Pengfei Liu, Xiangyang Xue, Zheng Zhang

摘要:

在本文中,我们引入先验知识,多尺度结构,自注意模块。我们提出了一种多尺度的Transformer,它使用多尺度多头自注意力来捕获不同尺度的特征。基于语言学的观点和对大规模语料库上预训练的Transformer(BERT)的分析,我们进一步设计了一种控制各层规模分布的策略。三种不同类型的任务(21个数据集)的结果表明,我们的多尺度Transformer优于标准的Transformer一致和显着的中小规模的数据集。

英文摘要:

In this paper, we introduce the prior knowledge, multi-scale structure, into self-attention modules. We propose a Multi-Scale Transformer which uses multi-scale multi-head self-attention to capture features from different scales. Based on the linguistic perspective and the analysis of pre-trained Transformer (BERT) on a huge corpus, we further design a strategy to control the scale distribution for each layer. Results of three different kinds of tasks (21 datasets) show our Multi-Scale Transformer outperforms the standard Transformer consistently and significantly on small and moderate size datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑